# Changelog

All notable changes to this project will be documented in this file.

The format is based on Keep a Changelog and this project adheres to Semantic Versioning.

Unreleased

Fixed

• missing legalForms trapped in atoka extractor

2.0.0

Changed

• contextual deepening on BandiCovid added
• contextual deepening mechanism implemented in Atoka's tasks and ETL classes
• django_uwsgi_taskmanager increased to 2.2.13 in requirements.txt (restart_despooled_tasks task added)
• spooler processes increased to 4 in uwsgi command line launch
• atokaconn external module replaces the atoka.connections package
• opdmetl replaced with ooetl (code refactoring)
• atokaconn 0.1.7 in requirements, fixes some issues with the connection to atokaconn

Fixed

• role of Revisore unico società per azioni added to allowed roles in atoka import; closes #374
• organizations part of the stock exchange are now aligned from Atoka; closes #457
• option and argument private_company_verification added to import, loader classes, and the org_lookup function, in order to force the check that organizations are not "private", before returninf a positive lookup result in lookup mixed_strategy; closes #458
• obfuscated results removed from the atoka extractor inner functions, as they missed packages (base, shares, ...) and generated errors in the import sub-tasks
• some logging improved

1.6.0

Added

• scripts to import commissioni bicamerali created;
• scripts to import commissioni refactored, so that a common logic (in SparqlToJsonCommand) is used;

Fixed

• parenthesis in names are escaped in anagraphical search utility (#166)
• criterion to skip repeated memberships in commissioni senato corrected (missing memberships are now imported)
• criterion to skip import of bicameral commissions from Senato corrected
• query to fetch Camera organs _(comissioni and uffici) corrected (missing organizations are now imported)
• appointment rules generation uses priority correctly
• appointments rules are generated correctly for sub-commissari in ASL (#447)
• appointable and appointer flags are visible in role_type serializer
• query to extract roles in minint historic scraper improved (administrative roles excluded)
• items to add or to remove only sent to output when starting dates differ more than 30 days
• django 3.2.6 in requirements.txt (safety fix)

[1.5.4]

Changed

• pydantic upgraded to 1.7.4
• django upgraded to 3.2.4
• tmp_list field renamed to tmp_lists to store connections to the temporary lists descriptions in memberships.

1.5.0

Added

• script to connect opdm entities (persons and lists) to electoral results implemented
• /memberships/ID/electoral_list_results endpoint
• /list_results/ID/memberships endpoint

Changed

• superfluous or deprecated tasks removed

Fixed

• dates mismatches fixed for memberships for commissions at la Camera, fetched from sparql endpoint
• parent_id changed into parent, in spqrql_meta intermediate json files, so that it is imported correctly in organizations
• founding_date and dissolution_date for a group in Senato are bound to the legislature

[1.4.3] - 2021-05-12

Fixed

• Algorithm fixed to avoid str to None comparisons

[1.4.2] - 2021-05-12

Fixed

• Algorithm filtering the results of gruppi at Senato adjusted, so that all groups are fetched properly

[1.4.1] - 2021-05-07

Fixed

• Sparql extraction corrected, so that the correct numbers are extracted
• Organizations and roles renamed, to match uniform naming and role types

1.4.0 - 2021-05-04

Added

• modular import task for commission and groups from Senato and Camera sparql endpoints corrected

Changed

• Django 3.2 compatibility (switched to popolo@django3 branch in requirements, too)
• candidate_membership serialization improved
• political_colour removed from serialized ElectoralCandidateResult (and tests)
• ordering for electoral results API improved
• redirection of electoral institution to organization
• redirection of electoral constituency to area
• redirection of electoral event to keyevent
• REST API max page size increased to 500
• local settings allow un-authenticated access to API
• electoral lists imported even if detached from candidates; pydantic model changed
• ElectoralCandidateResult is linked to a Membership, not a Person;
• ElectoralListResult may be linked to one or more memberships, but only in case of a plurinominal or proportional college in national elections;
• election models corrected: the ElectoralCandidateResult is linked to a Membership, not a Person; ElectoralListResult may be linked to one or More memberships, as well, but only in case of a plurinominal or proportional co
• ElectoralListResult.parties field in admin (uses autocompleter), to link one or more parties (Organization) to an ElectoralListResult instance
• forked easyaudit used in requirements (compatible with Django 3.2 added in setup.py)
• custom project.audit app extending easyaudit removed (back to simple)

Added

• elections endpoint in API for autocompleter's and smart OPDM UI
• elections ElectoralMembership class extends popolo.Membership with electoral results links for future reconciliation scripts
• name added to ElectoralCandidateResult serializer
• logging messages added to election results loader

1.3.5 - 2021-04-02

Added

• election results models, management tasks, api viewsets and serializers, admin section and tests (#456)

1.3.4

Fixed

• export_geojson task only extracts current identifiers, this fixes a issue on openpolis/geojson_italy

Changed

• /areas endpoint show identifiers start and end dates in inlines

1.3.3

Added

• Task to import world geometries, with dependencies from various sources;
• Task to import electoral constituencies' geoms for 2018, from ISTAT.

Changed

• django-popolo release upgraded to 3.0.2 in requirements.txt

1.3.2

Changed

• Migrations reset: the only dependencies from external dependencies can be the __initial migration.

Fixed

• Tests fixed

1.3.1

Fixed

• Percentages of ownerships are always rounded before insertion in DB and during verifications from ATOKA; this is needed to solve partially #455, as data from ATOKA come with varying floating representations of the same values;
• The script to consolidate ownerships, removing double values and incongruent dates has been improved; the easy_audit signal logging post_delete events has been disconnected within this script, as it generated errors, probably due to how easy_audit handles nested transactions.
• django-popolo package is installed from the requirements (pypi), not from gitlab's URL, so, the hack to check for changes is not needed in the Dockerfile and has been removed.

1.3.0

Changed

• All models related to electoral results removed from popolo, along with views and tasks, due to re-writing of the whole stuff.

Fixed

• organizations filtered for alignment with ATOKA now considers bot owner and owned organizations
• org deletion trapped in exception
• align atoka data task corrected: active parameters removed; doc improved

1.2.3

Added

• script to check overlapping and incompatibility of unique roles in local administrations
• persons endpoint in API now shows classifications for list and detailed view; classifications may be passed in the json for POST (creation) and PUT (update) operations;
• PopoloPersonWithRelatedDetailsLoader handles json of persons with classifications, same syntax of organizations

Fixed

• histadmin update logic improved: overwriting existing values is avoided; less duplicates are found
• typo in script_check_overlapping_unique_roles management task

1.2.2

Changed

• API read/write permissions granted to users in the opdm_redazione group

[1.2.1]

Fixed

• testuser need to be in opdm_api_writers group

[1.2.0]

Changed

• Class HasuraTokenObtainSlidingSerializer was renamed into CustomTokenObtainSlidingSerializer, in order to account for the fact that both hasura and opdm custom claims are handled.
• Same for CustomTokenObtainSlidingView.
• API permissions changed:
  • writing access is granted to all users in the opdm_api_writers group
  • reading access is granted to all users in the opdm_api_readers group
  • anonymous users do not have access to the API data any more
• udates of OPDM memberships coming from the scrape_histadmin_minint tasks do not overwrite existing values

Fixed

• management task script_correct_comuni generate an intelligible error when an Area cannot be associated with an Organization of type Comune;
• The special case "Comune Perfugas" is now handled well, which is relevant due to the particualrity of Sardinia's language;
• script_correct_memberships_overlapping_apicals.py fails with a trapped exception if no context is passed.

Added

• maintenance management tasks to associate correct electoral events to memebrships and correct end dates, in order for them not to cross next electoral event

1.1.18

Added

• topics app handles mapping between ATOKA_ATECO classifications and Organizations' topics
• labels app handles mapping between RoleTypes and OPDM_PERSON_LABEL Classifications and between FORMA_GIURIDICA_OP Classifications and OPDM_PERSON_LABEL Classifications
• Mappings can be imported from CSV sources
• Management tasks to sync OPDM_*_LABEL with RoleTypes or Classifications
• Signals handle detailed sync on Organizations and Persons, whenever classifications or memberships change

Changed

• CacheArgumentsCommandMixin methods names changed; explicit use required in classes using the mixin
• management tasks to automatic import labels from csv mappings
• migration to handle min year validator in OrganizationEconomicsHistorical
• task to consolidate ownership, updating dates or removing duplicated records implemented

1.1.17

Changed

• akas and economics parameters added to import_persons_all meta-command
• first row excluded from parsing while importing persons interests from remote file
• import of persons interest from remote file has now a dedicated import_persons_all meta-command
• clear_diff_cache and use_atoka_cache passed globally to import_atoka_all are used in importatoka* sub-commands
• data_path and batch_size variables renamed along all atoka management tasks
• datapath option passed to call_command for import operations, so that the given datapath is used, avoiding the default one (that generates errors in staging and production environments)
• correct logging to stdout added to call_command in import_atoka_all

1.1.16

Added

• meta management task to launch all of ATOKA-related ETL operations
• opdm_ids_file parameter added to import_atoka_persons script to only fetch data for given persons
• Script to generate personal relationships in neo4j implemented as management task
• Built-in audit log feature (based on easyaudit)

Changed

• django-uwsgi-taskmanager upgraded o 2.0.4 (management tasks return result)
• script that merge duplicates organizations adjusted
• parameters added to customise script_check_discrepancies;
• urls to affected entities added in log messages;

1.1.15

Changed

• import_orgs_from_json refactored, to use JsonDiff class;
• import_ownerships_from_json refactored, to use JsonDiff class;

1.1.14

Added

• ETL procedures from ATOKA refined.
• It is now possible to import json data differentially.
• JsonDiffCloseItemsInRemovalsCompositeETL allows differential import of items that are closed when missing elements are detected in diffs.

Changed

• The script to merge organizations now merge the memberships, before removing 'old' organizations.

1.1.13 (YANKED)

Added

• Script to transfer person's headshots to S3 implemented.
• Import of information from atoka, starting from persons in OPDM is now possible.
• Person and Organization appear now in the admin site.
• Thresholds for AKAs can now be defined dynamically, as management task's parameters.
• v1/memberships now accepts PATCH request, enabling the (partial) update of a whole batch of `Membership objects.
• Add Interval validator to be used when serializing models with start_date and end_date (Dateframeable).

Changed

• Pool of level-0 organizations restricted: MIUR organizations removed.

1.1.12

Fixed

• shareholders imported together with sharesOwned organizations; shares-level=0 import all level-1 orgs, and their shareholders;

1.1.11

Fixed

• transformation producing organizations json file corrected; owned orgs and shareholders added also when main org already added; this avoids missing organizations log messages;

Added

• fullAddress property from atoka imported into all Organizations, as contact_detail of type MAIL.

1.1.10

Added

• Import shareholders along with sharesOwned from atoka.

1.1.9

Added

• script that computes participation level starts with broader criterion, setting all organizations having classification TIPOLOGIA_IPA_BDAP set to PA
• atoka extractor considers additional CFs in identifiers with scheme ALTRI_CF_ATOKA, when creating the batches of tax_ids to fetch
• atoka extractor extracts shares even when percentage is null;
• organizations lookup in opdm core OrganizationUtils looks for identifiers within ALTRI_CF_ATOKA identifiers

1.1.8

Added

• ATECO classifications for organizations are imported from atoka

1.1.7

Added

• script to compute public interest and classify participation level
• meta script to launch all upgrade tasks from atoka implemented

1.1.6

Changed

• mechanism to fetch all results from atoka, exceeding the maximum limit improved

1.1.5

Fixed

• bug on merge of same tax_id info in different atoka records, during extraction corrected;

Added

• macro management task composing atoka imports implemented

1.1.4

Added

• pub_part_percentage field to OrganizationEconomics model.
• • --shares-level argument added to import_atoka_extract_json task, defaults to zero

Changed

• min and max validators corrected for revenue_trend
• a limit greater than 50 may be specified in the get_items_from_ids atoka connections method
• AtokaOwnershipsExtractor fetch information about owned share also from non-active companies

1.1.3

Fixed

• script to import organization from BDAP uses the correct attribute (source, not remote_url)

Added

• script to correct duplicates of some institutions (regioni and comuni)
• API /organizations URLS now show and can be filtered/ordered by main economics indicators: employees, revenue, capital_stock

Changed

• atoka_persons_memberships.json is now composed of unique Persons with multiple Memberships optimising number of similarities that will be created and avoiding double similarities

1.1.2

Fixed

• founding_date is imported correctly
• double FORMA_GIURIDICA_OP classification is avoided when importing from atoka
• RoleTypes generation from atoka mapping corrected
• Commissario straordinario removed from atoka_roles_skip_list

Changed

• CCIAA classification and REA identifier from atoka moved into one CCIAA-REA identifier
• It is now impossible to add more than one classifications of the same scheme to an object (popolo).

1.1.1

Added

• management task to import OrganizationEconomics, with historical records, from atoka

Changed

• is_public flag (SPA) and capital_stock added to OrganizationEconomics
• json files for organizations can specify classifications by: classification (id), scheme/code, scheme/descr, scheme/code/descr (non-existing)

1.1.0

Added

• project.atoka module added, to handle all atoka-related logic and tasks
• OrganizationEconomics model added to atoka module

Changed

• Unofficial roles are no longer imported from atoka.
• atoka-related code moved into project.atoka module (which is a Django app).

1.0.3

Changed

• contexts may be specified for atoka transformations in import_atoka_transform management task, so that now the complete atoka import procedure described in the gitlab snippet (<https://gitlab.depp.it/openpolis/opdm/opdm-service/snippets/26&gt;), can be launched

1.0.2

Changed

• --use-dummy-transformation added to import_persons_memberships_from_json task to keeb back compatibility with national level imports and use it in new atoka imports
• --log-step and --context parameters added to import_persons_memberships_from_json to allow for setting context string in loader_context (similarities)

1.0.1

Added

• ETL tasks for atoka integration
• maintenance script for organizations missing classifications
• maintenance script for duplicate organization merge
• data changelog and main changelog urls and views added and linked to in about page
• verification of administrative history of an institution from minint

1.0.0

Added

• import of ownerships from atoka's API
• /keyevents endpoint added in API, to handel KeyEvents
• /organizations API endpoint manages the key_events relations
• organizations loader created under etl/loaders
• JSONArrayExtractor created under etl/extractors
• project.api_v1.etl.transformations.json2opdm package added with ETL classes dedicated to processes reading JSON content into OPDM (parsers)
• memberships/ PUT, POST and PATCH handle parameters from UI (#76)
• import_persons_memberships_from_op tasks implemented accepts parameters to:
  • import from different institutions
  • handles pagination and limits
  • handle updates and full imports
• Op2OpdmETL implemented into etl.transformations; Handles different transform for persons and memberships, needs to be used with the low-level interface.
• n_memberships and n_active_memberships computed fields added to organizations list
• classification_id filter added to /organizations endpoint
• Script to remove organs with no memberships in organizations having more than 2 organs
• AKA api and tests implemented
• akas model implemented
• import_minint_year task to import from Minint yearly upgrades
• role_types endpoint added to API
• import role_types from google docs
• import_orgs_from_bdap management task revised
  • new columns from BDAP csv file
  • forma_giuridica_op buil rules implemented
• akas app added to project (to manage similarities)
• solr added to stack to implement search endpoints
• /memberships endpoint accepts now write operations

Changed

• solr index changed in order to handle textual fields as single words, using no unicode (diacritics); text_with_diacritics field type introduced in schema.xml;
• solr deploy changed, default dir is now /opt/solr/server/solr/mycores/opdm
• import_op_location_id_from_op has an --api-filters option, to restrict the field of application of the procedure
• import_orgs_from_bdap refactored; import utils moved to core.py; lookup and update strategies are now parametric.
• data structure for get_organsdict* core methods get a consiglio_metropolitano key, disambguating from consiglio, that refers to consiglio provinciale, when the two are present for the same op_id, or istat code, or province id.
• script to add Government and Council organs to local institutions corrected; it now provides the founding_date and dissolution_date, instead of start_date and end_date.
• update_strategy can now be specified for memberships import operations
• CF is computed and added in a post_save signal handler
• import_persons task now accepts persons-update-strategy arguments
• memberships loader now accepts upload_strategy (keep_old)
• AKAs hi and lo thresholds defined in settings
• core package refactored, all logic shared in imports is here
• org_id and org_scheme added to dataframe for Organizations
• PopoloPDLoader removed and unified into PopoloLoader
• refactoring ETL classes moved into etl.transformations
• deleted old and unreliable management tasks to import from OP; exception trapping in MembershipLoader improved
• PopoloPersonLoader and PopoloMembershipLoader added to separate loading phase;
• OPAPIPagedExtractor added to available extractors
• build_charge_descr_from_opapi_data method added to core.LabelsBuilder class;
• update_or_create_person_from_item method uses DB transactions;
• synch with solr is handled at low level, in a transactional way;
• search_person substituted with strategy-specific methods;
• PopoloPDLoader added to differentiate from PopoloLoader. PopoloPDLoader uses Pandas as source of the data to load, while PopoloLoader uses a generic iterator.
• All persons identifiers are added to solr index (as string). Fields name are extracted from the scheme (OP_ID_s, ...).
• HAYSTACK_SIGNAL_PROCESSOR value is read from env. Defaults to BaseSignalProcessor, so that manager tasks importing data must handle DB-solr synchronization at low level.
• import of minintakas from OP creates AKA objects
• memberships import management task renamed
• minintakas import management task name changed
• Logging for RoleType.DoesNotExist exception improved
• exact_name and parent_identifier filters added to areas endpoint
• Search accepts birth_location_area as parameter (integer)
• Profession and Educational levels serializers and views added
• About page links to documentation corrected; swagger is now used
• Organization filters corrected and improved
• Django version pin-pointed
• When comparing objects returned from a POST or PUT request, complex objects (Person, Area, ...) have an id in the original request, and returns a dictionary.
• Posts and Memberships relations adjusted to reflect popolo standards
• All references to rancher removed from docs
• Posts and Memberships changed in imports
• write API (creation and update), implemented for Person, Organization and Area
• django-popolo and django-etl git urls commented in requirements.txt, as they're installed separately in the docker image build process. Developers need to install them manually.
• Organization, Person and Area writer serializers create and update methods collected in a dedicated mixin.
• detail_route and list_route decorators substituted with action,
• autocompleter custom action added to areas
• AreaAutocompleterSerializer serializer implemented;
• id is now explicitly shown in Generic Relation Serializers

Fixed

• HAYSTACK_SIGNAL_PROCESSOR set to disabled default processor, so that tests do not raise errors generated from missing solr in test environment
• Data-migrations added to correct errors generated by organization imports;
• typo in Task post_save signal corrected
• tabular.html template now handles the case of null original.ct_id

0.9.0

Added

• former_parents field added to /areas/ID/ detail API view
• /areas/ID/former_children view added to areas API view

Fixed

• wrong migrations dependency on django_celery_beat removed from migrations

0.8.0

Added

• tasks_manager app implemented

Changed

• pipeline deploy is manual, after pushes to master
• redis panels removed in tmux session

0.7.2

Fixed

• mistune library added to requirements

0.7.1

Fixed

• popolo.urls neede by tests added to main urls

0.7.0

Added

• home page redirects to /about page
• /about page shows useful links for developers
• /about page contains a detailed description of the project's technical details, and basic statistics on data
• markdown content within django templates are now possible, through markdown_tags.markdown template tag

Fixed

• bug in locations extraction with limit and offset fixed

Changed

• gitlab CI deploy to production added (master only)

0.6.0

Fixed

• problem in REG context for posts_and_persons import solved

Changed

• offset parameter added to import_posts_and_persons_from_op management task, useful in REG and COM contexts.

0.5.2

Fixed

• get_or_create criterion while creating global organizations corrected

0.5.1

Fixed

• one label in Post was corrected, for posts imported from openpolis

0.5.0

Added

• api endpoints, documentations and coreapi schema implemented for:
  • areas
  • organizations
  • persons
  • memberships
  • posts

Changed

• Local Administration import refactored, so that the locations are fetched from Area, not requested to openpolis api
• location-limit parameter added to management task the number of locations processed for memberships import can be limited by this command line parameter

0.4.0

Added

• Senato della Repubblica import added to it import context
• Camera dei Deputati import added to it import context
• Local Administration import added, with:
  • locations fetch from API
  • locations organizations setup
  • dedicated labels construction

Changed

• url_filters can now be passed to OpAPIImporter.import_memberships
• constituency_descr and electoral_list_descr fields added to Membership in order to store relevant information contained in the Openpolitici dataset

0.3.2

Fixed

• January 1st birth dates are emitted as warning, as they're used by MinInt when detailed information are missing.
• Multiple overlapping memberships are possible if the allow_overlap flag is specified
• When memberships refers to overlapping dates a warning is emitted

0.3.1

Fixed

• labels for Posts are now separated from the extended descriptions while importing charges from Openpolis

0.3.0

Added

• import for persons from openpolis API started

0.2.0

Added

• REST API endpoints for areas and organizations

Fixed

• deploy in staging only happens after pushes to develop branches
• instructions on data import improved

0.1.3

Fixed

• bumpversion configuration file added

0.1.2

Fixed

• Import of organizations from BDAP now reads CSV from AWS s3
• Project name and version added to settings and admin templates

0.1.1

Fixed

• all tests now invoked using spatialite database; libspatialite5 added to Docker image; popolo and etml tests invoked in the test stage.

0.1.0

Added

• first release: areas and organizations import management tasks added