European Commission logo
INSPIRE Community Forum

It seems that there is only one fully compliant solution to serve multiple harmonised datasets trough WFS 2.0

We investigated several solutions to serve multiple harmonised datasets trough WFS 2.0 according to Technical Guidelines for Download Services. We are not discussing about providing access to only one dataset per server, but to provide access to more datasets.

 

Update 8 September 2016: The single compliant solution seems to be Snowflake GoPublisher WFS.

 

Update 9 May 2018: Because Stefania pointed in this tread that that Deegree is able to serve two different datasets for the same XML schema at different endpoints (asking for typeNames=ps:ProtectedSites in both cases) we started some tests.

So, it is still to see if Deegree is able to serve two different datasets based on multiple XML schemas at different endpoints in order to say that Deegree is compliant with Requirement 52, but is a big step forward in fullfiling the Requirement 52. For those data providers that need to serve feature types from one application schema at one endpoint Deegree is fulfiling the Req 52. We did not tested it yet to see if there are not other issues in order to confirm that it is able to serve multiple harmonised datasets based on multiple XML schemas trough WFS 2.0 according to Technical Guidelines for Download Services. Versions 3.4-RC (unstable) are passing the EPSG in URI format test, while 3.3 (stable) are not passing the test as EPSG is provided in URN format. As Deegree versions 3.4-RC are advertised as unstable and as there is no 3.4 stable version, it is not sure if Deegree is passing all the requirments of the Technical Guidelines for Download Services. However it should be noted that Deegree 3.4 version is OCG WFS 2.0 certified. If anyone had tested Deegree against each requirement from the TG, please share this information.

To conclude about Geoserver 2.13.0, for those data providers that need to serve feature types from one XML schema at one and only one endpoint, Geoserver 2.13.0 is fulfiling the Req 52. Geoserver seems not to be able yet to serve feature types from the same XML schema  at different endpoints, even if using the newly created isolated workspaces.

 

Till 8 September 2016 we thought that Go Publisher WFS (version 4.0.2) is not fullfiling the TG Requirement 46 ”Implementations shall conform to ISO 19142 Conformance Class Simple WFS”, more precise the WFS DescribeFeatureType request is not providing the expected result. Daniel Cocanu discovered that this is not related to GoPublisher WFS but it is related to XSD version 3.0 schemas.

 

The second most compliant solution seems to be Geoserver.  Unfortunately it does not fulfill at least one requirement, namely Requirement 52 ”A separate WFS endpoint shall be provided for each INSPIRE dataset thus providing one dataset per GetCapabilities response.” Tested on version 2.8.3. 

On third place seems to be Deegree (very good for testing purposes as it alows to serve datasets from the GML stored in memory) and on forth place ESRI ArcGIS for INSPIRE. We did not tested any other solution and we do not know any other solution (probably FME from SAFE can be a solution as well). 

Does anyone knows how to provide separate WFS endpoints for each INSPIRE dataset with Geoserver? 

Does anyone know other solutions to be tested (i.e.: FME from Safe) ?

Deoes anyone know other INSPIRE TG Requirements or Recommandations that cant be fullfiled by existing solutions ?

Isnt'it quite strange that Technical Guidelines are setting requirements that cant be fullfiled by any existing solution ?

Best regards,

Iurie Maxim

  • Ilkka RINNE

    Hi Iurie, all,

    Thanks for this interesting discussion!

    What comes to WFS implementations supporting INSPIRE requirements, we did some research on this at the Finnish Environment Institute in autumn 2015, and were surprised to find that while the Snowflake Go Publisher WFS otherwise seemed like one of the best solutions out there, there was no easy way to automatically keep the WFS capabilities documents up-to-date if we also wanted to include the INSPIRE extended service metadata.

    The only possibility to make the Go Publisher WFS service INSPIRE compliant was to manually create and maintain the capabilities documents, which can then be externally referred to from the Go Publisher created WFS server. This was a major issue for the Finnish Environment Institute, as manually editing the capabilities had proved to be a major source of extra work using the Esri ArcGIS Server based solutions.

    We also noticed the null-namespace problem of Geoserver/AppSchema plugin, as well as the inability to run two Geoserver workspaces containing the same feature types within one Geoserver instance. If the GS workspaces were not bound to the namespace prefix of the schema, the requirement 52 could be much more easily implemented using Geoserver. In many cases several INSPIRE data sets share the same feature types, but several the GS workspaces (with WFS endpoints) with the same provided feature types cannot co-exist within the same GS server instance.

    As far a I understand the main reason for the Requirement 52 was that without it, it was not possible to map the mandatory INSPIRE Describe Spatial Dataset operation to a WFS service. It's now mapped to GetCapabilities operation of a WFS service (see page 27 of the TG Download Service 3.1), and thus the returned capabilities document shall only contain description of a single dataset, thus Requirement 52. 

  • Katharina SCHLEIDT

    By Katharina SCHLEIDT

    Hi all,

    some more thoughts on this:

    On the one side, I'm wondering if we couldn't mess with requirement 52 a bit. While not totally formally corrent, at least I've always worked on the theory that commission is better than ommission. I totally understand the issues arising, had enough fun searching for the correct featureType in bundled services. However, this could easily be overcome by providing information on the featureTypes pertaining to a dataset in the metadata, problem solved on a functional level. Worth trying?

    On the other side, I'm wondering how far we could get around this problem with some URL-rewriting? We'll need this type of functionality anyway if we want to make the xlinks resolvable (see the discussion under https://themes.jrc.ec.europa.eu/discussion/view/21204/inspire-and-referencing). If we put up a rewriting rule that cuts the namespace from requests containing a getFeature we should be clean - or am I missing something?

    :?

    Kathi

  • Iurie MAXIM

    Update 8 September 2016:

    Daniel Cocanu discovered that the issue with DescribeFeatureType is not related to GoPublisher or Geoserver. The DescribeFeatureType does not work on XSD 3.0 versions schemas, but works on XSD 4.0 versions schemas. This is due to the fact that the links provided in the targetNamespace are not resolvable in 3.0 schemas:

    targetNamespace="urn:x-inspire:specification:gmlas:ProtectedSites:3.0" version="3.0"

    targetNamespace="http://inspire.ec.europa.eu/schemas/ps/4.0" version="4.0">

    Therefore we closed the CASE 00001948. 

    We changed as well the title of the post into "It seems that there is only one fully compliant solution to serve multiple harmonised datasets trough WFS 2.0"

  • Ilkka RINNE

    I would still argue that the while formally Go Publisher WFS can be used provide a compliant INSPIRE WFS Download Service (predefined and direct access), it's support for it is quite weak: There is no mechanism to provide any on the required INSPIRE extended capabilities, except by manually writing a static capabilities document file each time the service is changed.

    Also there is no multi-language support in Go Publisher WFS. This is not strong point Geoserver either, but at least it "supports" using a single declared language (with the help of the INSPIRE extension).

  • Iurie MAXIM

    Hi Ilkka,

    Indeed in order to provide INSPIRE extended capabilities with Go Publisher WFS it is necesary to manualy edit the GetCapabilities XML document in Notepad++ (or simmilar).

    But this is true as well for the metadata of the Dataset and for the metadata of the Download Service, when the service is changing.

    Therefore I do not understand clearly the issue, as far as we need to change three XML documents when the service is changing, not only the GetCapabilities document, but the metadata files for the dataset and for the download service (for view service as well).

    While testing almost all of the available solutions for creating WFS 2.0 compliant services we found that Notepad++ is necessary in all circumstances. For example we used Notepad++ a lot for App-schema for Geoserver as HALE does not support all the necesary mappings (i.e.: providing upper level administrative units for au:AdministrativeUnit feature types). We used Notepad++ even more for Deegree. We used Notepad++ for creating XML metadata documents for dataset and for data services as we did not find any Metadata editor able to provide all the metadata elements (i.e.: those in Annex B of MD IR).

     

    Regarding multi-language, sincerly I expect first of all that the EC Metadata Editor to allow the creation of multi-linguage XML Metadata Documents. Curently Notepad++ is the solution for creating such metadata documents and I saw, but not tested a solution from Intergraph.

     

    I can say as well that the Go Publisher has an very old style interface reminding me of FoxPro, not designed for usability (i.e.: each time is created a new element it needs to be put in his position whithout having any clue where it should be and you need to right-click on it and select move up then the element is automaticaly unselected, so the right-click must be repeated this until the element is aranged at his guessed position). Compared to HALE usability and interface, Go Publisher is years behind.

    But fortunately Snowflake Software chosed for Go Publisher WFS to store the mapping documents in separate folders, one folder for each maping document, allowing therefore to create multiple WFS services at multiple endpoints, each WFS Service beeing able to provide access to one dataset with one or many feature types. I did not seen any other solution that is able to do this. Due to this, Go Publisher WFS can be used to serve multiple datasets, a function that is crucial for a data provider. I cant imagine a data provider that has only one dataset, therefore I do not know what other solution a data provider has to serve its datasets trough WFS.

    I appreciate a lot the information what you shared about your experience with Geoserver because we had simmilar findings and I did not had currage to clearly say that Geoserver cant be currently used by a data provider for delivering INSPIRE WFS Download services in practice, but now I can say this clearly. As well, I dont think that Geoserver will be changed soon so dramaticaly in order not to have anymore one workspace for each XSD schema, this beeing the major issue that makes Geoserver not suitable for INSPIRE implementation.

    Unfortunately Geoserver is a big dissapointment for me while looking at all efforts made for HALE and App-schema develpoment. I just hope to be wrong.

    Regarding Requirement 52, I want to thank you for sharing the information of its necessity  and I have a simmilar understanding that Requirement 52 is a must for linking Metadata to the WFS Service,  as I explained in a previous post within this discution when I inserted this picture:

     

    On the other hand I do not clearly understand what dynamic has the dataset for which you needed to manualy create the GetCapabilities document and what parts of the XML needs updates each time the service is changed. I would appreciate if you can provide more details. Those changes do not require as well the change of the Dataset MD and of the Service MD (that are done manualy as well) ?

    Best regards,

    Iurie Maxim

  • Iurie MAXIM

    Hi Kathi,

    The solution for the problem that you described are stored querries. Stored querries can be created in order to retrieve the features by their inspireId. For those feature types that have no lifeCycle, the stored querries must filter only by localId. For those feature types that have lifeCycle, the stored querries must filter by both localId and versionId. There is no need to filter by inspireId namespace.

    After making these stored querries the link for retrieving a feature is much more simple and can be more easily used for redirecting.

    In order to be used for redirecting, we named all these querries in a standardised way like: GETpsPSByInspireId, GETauAUByInspireId, GETauABByInspireId, GETbrBRByInspireId and we are still asking ourselves if we should name the stored querry(ies) for retrieving NamePlaces either as GETgnNPByInspireId or we should make separate stored querries for each NamePlace type, like NamePlace for psPS and NamePlace for auAU.

    You may have a look at this post as well to better understand. I explained there how are looking the links that are used for redirecting :

    http://gmlid.ro/ENV/PADS/psPSRef/RONPA0022 ->

    http://inspire.biodiversity.ro/WFS/RO_ENV_PADS/wfs?service=WFS&version=2.0.0&request=GetFeature&STOREDQUERY_ID=GETpsPSRefByInspireId&localId=RONPA0022

    I am not sure if with KEMP we are able to make this kind of requests http://gmlid.ro/ENV/PADS/psPSRef/RONPA0022, but most probably these are easier to implement: http://gmlid.ro/ENV/PADS?psPSRef/RONPA0022

    For those feature types having lifeCycle, the requests can look like

    http://gmlid.ro/ENV/PADS/auAU/HD/2016-07-11 or easier http://gmlid.ro/ENV/PADS?auAU/HD&2016-07-11

    auAu is the au:AdministrativeUnit feature type, "HD" is the localId of one administrative unit and 2016-07-11 is the version of the feature having the localId "HD" that has the endLifeSpan from date 2016-07-11. We cut the time as ":" character is not suported in versionId element. If it is necesary to store the time as well, the character ":" must be replaced with other suported characters, for example "-".

    http://gmlid.ro/ENV/PADS/auAU/HD would be the request for the au:AdministrativeUnit feature type (with lifeCycle) with the localId = "HD" that has no version, namely that one that is curently valid, so that one that has no endLifeSpan element at all.

    Note 1: psPS stands for ps:ProtectedSite feature type, brBR for br:Bio-geographicalRegions, auAB for au:AdministrativeBoundaries, etc

    Note 2: When you mentioned ”namespace” I suposed that you are refering to InspireId namespace and not to other namespaces.

    Best regards,

    Iurie Maxim

  • Katharina SCHLEIDT

    By Katharina SCHLEIDT

    Hi Iurie,

    the stored queries are a further (undocumented) piece of the puzzle (and one of the many little bits and pieces that should also be standardized to assure interoperability), together with the rewriting for xlinks.

    But I was thinking about the Requirement 52 bit here. Wouldn't the following work:

    Set up GeoServer for all themes/datasets on one base URL: www.datasource.eu/geoserver/ows?service=WFS...

    Do a rewrite rule for the various theme/dataset specific URLs, i.e. www.datasource.eu/geoserver/DS/ows?service=WFS... back to the base URL above (whereby DS is the theme/dataset of your choice)

    The client sees unique endpoints per theme/dataset

    GeoServer can happily & autistically continue to serve one end-point

    I've never used rewriting myself, but i'm assuming that the various parameters (service=WFS&version=...) can be passed on in the new rewritten request.

    :?

    Kathi

    PS: thanks for making me feel less stupid about also using Notepad++ as my main configuration tool! ;)

     

  • Iurie MAXIM

    Hi Kathy,

    I understand that you take into consideration only the case where one dataset = one data theme.

    Namely you consider that a dataset exist for example for ps:ProtectedSite and another dataset exist for example for br:Bio-geographicalRegion.

    If looking at INSPIRE IR, especialy at the MD IR it is clear that a dataset can contain more than one data themes, namely more than one feature type.

    So a dataset can contain both psPs and brBR.

    Article 8.1 of the INS DIR is mentioning: "In the case of spatial data sets corresponding to one or more of the themes listed in Annex I or II...". INS DIR defines ‘spatial data set’ as an identifiable collection of spatial data;

    MD IR clearly specify that if there are data at different resolutions/precisions, then different datasets must exist (i.e.: 1:200.000 Geolology national coverage and 1:5.000 Geology for smaller areas). So there must be two GE datasets.

    Sincerely most data providers are in this case having different feature types that are fulfiling certain topology rules (i.e.: common boundaries between PS, BR, AU, HY, etc) and several data at different resolutions for diferent areas (i.e.: GE, LC, SD, HB).

    Working a lot with ArcGIS my understanding is that:

    one dataset = one ESRI Feature Dataset 

    My coleagues knows that one year ago I had the same understanding as yours, namely:

    one dataset = one feature type or at least all feature types from one data theme

    (For example AU data theme has two feature types au:AdministrativeUnit - poligon and au:AdministrativeBoundary - line).

    Geoserver has the same approach as yours, namely one dataset = one feature type schema as it can be seen here: http://docs.geoserver.org/stable/en/user/services/wfs/schemamapping.html

    But this is not according to INSPIRE and not according to ISO definition of a dataset.

    In order to be able to use this aproach with Geoserver in INSPIRE, I think that we needed to have only one inspire_features.XSD schema containing all feature types.

    Therefore my understanding is that Geoserver can serve only one INSPIRE dataset with one feature type or with multiple feature types but the data provider needs to strugle with the GN common to multiple schemas. Geoserver is not able to serve multiple INSPIRE datasets, Geoserver is not able to serve two datasets of the same type  (i.e:two GE at different resolutions/precision)

    On the other hand, unfortunately the "null" issue is much more serious that we initialy thought and we found that Gesoerver has a real problem with these "nulls" for complex features in other requests as well.

    For example this request

    http://inspire.biodiversity.ro/geoserver/ows?service=wfs&version=2.0.0&request=GetPropertyValue&typename=gn:NamedPlace&valueReference=gn:inspireId&count=2

    returns ”null” instead of ”base”:

    <gn:inspireId>

    <null:Identifier xmlns:null="http://inspire.ec.europa.eu/schemas/base/3.3"&gt;

    <null:localId>ROMAB0001</null:localId>

    <null:namespace>RO.ENV.GN.</null:namespace>

    </null:Identifier>

    </gn:inspireId>

     

    Ilkka wrote as well about the "inability to run two Geoserver workspaces containing the same feature types within one Geoserver instance." and that "the GS workspaces" are "bound to the namespace prefix of the schema",

    The main issue will be with the gn:NamedPlace because many data themes are integrating as well the GN in order to provide the names of the features.

    Therefore a data provider will be able for example to provide the PS + GN (with the names of the protected sites), but will strugle to provide as well the HY + GN (with the names of the rivers) within the same Geoserver instance.

    To be more clear, if a dataprovider has a dataset containing for example:

    - the boundaries of protected sites as poligons (PS + GN)

    - the segments of the river embankments that are part of the boundaries of the protected sites (HY + GN)

    - the segments of the administrative boundaries that are part of the boundaries of the protected sites (AU + GN)

    - etc

    then that data provider will strugle to provide that dataset with Geoserver because of multiple GN for PS, HY and AU.

    If that data provider has for example as well another dataset with a national coverage with the rivers but at another scale less precise, he will not be able to use the same Geoserver instance to provide this dataset as well and he will need to instal Geoserver twice or use another server. This is because he needs two endpoints for the two GetCapabilties documents, but he has only one endpoint for all geoserver services and only one workspace (and endpoint) for HY.

    Simmilarly lets imagine a data provider that has geological data for diferent areass at different scales. Lets imagine it has a dataset with national coverage at 1:200.000 and another dataset more detailed but only for some areas, at scale 1:5.000. According to MD IR he will need to create two datasets and to specify in each MD document the precision/scale, that differs. He will not be able to serve these two datasets with the same geoserver instance because he will need two endpoints for each GetCapabilities documents, but Geoserver has only one workspace for each XSD schema (GE.XSD for geology), so he will not be able to use one Geoserver instance to provide these two datasets and he will need to install Geoserver twice on the same server or to use two separate servers.

    Now imagine the real situation of a data provider with many datasets. Can he use Geoserver. I would say clearly: No, he cant. Unless he wants to have many servers with many Geoserver instances and is able to manage all of them and to strugle with the GN for multiple data themes within the same dataset.

    Will Geoserver be changed in the near future to suport these real situations ? Dificult to imagine as it must be re-writen from its core.

    Toghether with the "null" issues, all these are making Geoserver unusable for serving WFS INSPIRE Download Services in a real environment.

    As I previously mentioned, Geoserver is a big dissapointment for me for implementing 
    INSPIRE, while looking at all efforts made for HALE and App-schema develpoment.

    I just hope to be wrong,

    Iurie Maxim

  • Katharina SCHLEIDT

    By Katharina SCHLEIDT

    Hi Iurie,

    many thanks for the ESRI Feature Dataset definition! makes arguing against the current Requirement 52 that much easier. If a Feature Dataset is defined as:

    "A feature dataset is a collection of related feature classes that share a common coordinate system. Feature datasets are used to spatially or thematically integrate related feature classes. Their primary purpose is for organizing related feature classes into a common dataset for building a topology, a network dataset, a terrain dataset, or a geometric network."

    One could argue that a data holders dataset is the collection of their INSPIRE data ;)

    ---

    As to the bit with GeoServer and namespaces, yes, at the end of the day, if you're running multiple themes which use the same base type such as GN, you have to serve the data from the GN data from a merged table consisting of the GN data from all of these themes; as to my view GeoServer should be serving data from dedicated tables (so transfered/transformed from the DB in which the data manipulation takes place to a DB only used for GeoServer) this shouldn't cause a problem.

    :?

    Kathi

  • Iurie MAXIM

    Hi Kathi,

    Regarding ESRI Feature Dataset, when creating it, an user must provide the coordinate sistem and the tolerance (that is linked to precision/scale and that almost no user is actualy changing).

    An user cant build topology for the feature classes within a feature dataset if they are not in the same precision/at same scale.

    "One could argue that a data holders dataset is the collection of their INSPIRE data" only if he can create a single metadata document for all these data. This means that all those spatial data must have the same lineage, that will be dificult to believe.

    Regarding GN table, in real implementations spatial views are used rather than tables.  Combining those GN tables in one spatial view seems not to be too simple and therefore I mentioned that a data provider needs to strugle. In a static example is simplier. In a database that is changing daily/periodicaly is more complicated.

    Iurie

This discussion is closed.

This discussion is closed and is not accepting new comments.

Biodiversity & Area Management

Biodiversity & Area Management

If themes like Protected Sites, Area Management/Restriction/Regulation Zones and Reporting Units, Habitats and Biotopes, Species Distribution, Bio-geographical Regions matters to you, join these groups!