What is it with Manifold, British National Grid projection and databases.

I've been trying to set PostGIS up and transfer some of the data held in Manifold map files into PostGIS.

However, if I have a drawing whose projection is set to British National Grid and export it to PostGIS, on the dialogue..see attached image...it states 'projections are different' and matches it to 'OSNI 1952 Irish National Grid'. British National Grid is EPSG 27700, which is listed in the PostGIS supported coordinates.

If I export a dataset to PostGIS using Cadcorp GIS, I know all the geometry is in the correct place, and I know definitely that it exports it to PostGIS using EPSG27700 (British National Grid). So why, when I then link to this data from Manifold does it bring it in with a transverse mercator projection as per second screenshot? As per the screenshot called 'postgis', the table has plainly gone into the database with EPSG27700 SRID so why does Manifold then retrieve it with a transverse mercator projection as opposed to British National Grid (which are slightly different)?

Attachments:
postgis.jpg
Projections.jpg
Projections2.jpg

Update....

I think Manifolds got it's ideas about 'datum' wrong.

As per the screenshot, I created a brand new drawing and gave it a British National Grid projected coord system. When I try and export that drawing to PostGIS, as per the dialogue in the screenshot, Manifold tries to match it with the OSNI Irish National Grid projected coord system. Underneath that it states 'projections are different'. When I click on the button to select another projected coord system, and select British National Grid, it states 'projections are exactly the same, but datums are different'.

So the question I've got is why's Manifold's British National Grid projected coordinate system different to the PostGIS British National Grid projected coordinate system?

Attachments:
PostGIS_projections.jpg

AFAIK Manifold is not using the name of a projection and I guess srid codes for export only. For import it compares the set of parameters with manifolds projections.

PostGIS in spatial_ref_sys offers

srid= 927700

auth_name='epsg'auth_srid=27700proj4text='+proj=tmerc +lat_0=49 +lon_0=-2 +k=0.9996012717 +x_0=400000 +y_0=-100000 ' & _ '+ellps=airy +datum=OSGB36 +units=m +no_defs 'srtext='PROJCS["OSGB 1936 / British National Grid,"' 'GEOGCS["OSGB 1936", 'DATUM["OSGB_1936", 'SPHEROID["Airy 1830",6377563.396,299.3249646 'PRIMEM["Greenwich",0 'UNIT["degree",0.01745329251994328 'UNIT["metre",1,AUTHORITY 'PROJECTION["Transverse_Mercator"], PARAMETER["latitude_of_origin",49],PARAMETER["central_meridian",-2] PARAMETER["scale_factor",0.9996012717] PARAMETER["false_easting",400000],PARAMETER["false_northing",-100000],AUTHORITY["EPSG","27700"] AXIS["Easting",EAST],AXIS["Northing",NORTH]]'

vs. Manifold

<preset> <name>British Grid</name>

<datum>Ordnance Survey Great Britain 1936 (England)</datum>

<system>Transverse Mercator</system>

<unit>Meter</unit>

<majorAxis>6.377563395999999700000000e+006</majorAxis>

<eccentricity>8.167337387414189100000000e-002</eccentricity>

<centerX>3.710000000000000000000000e+002</centerX>

<centerY>-1.120000000000000000000000e+002</centerY>

<centerZ>4.340000000000000000000000e+002</centerZ>

<rotationX>0.000000000000000000000000e+000</rotationX>

<rotationY>0.000000000000000000000000e+000</rotationY>

<rotationZ>0.000000000000000000000000e+000</rotationZ>

<scaleAdjustment>0.000000000000000000000000e+000</scaleAdjustment>

<scaleX>9.996012717000000200000000e-001</scaleX>

<localScaleX>1.000000000000000000000000e+000</localScaleX>

<scaleY>9.996012717000000200000000e-001</scaleY>

<localScaleY>1.000000000000000000000000e+000</localScaleY>

<falseEasting>4.000000000000000000000000e+005</falseEasting>

<localOffsetX>0.000000000000000000000000e+000</localOffsetX>

<falseNorthing>-1.000000000000000000000000e+005</falseNorthing>

<localOffsetY>0.000000000000000000000000e+000</localOffsetY>

<centerLat>4.900000000000000000000000e+001</centerLat>

<centerLon>-2.000000000000000000000000e+000</centerLon>

</preset>

Klaus, I think you've summed it up in a nutshell.

Manifold is correctly interpreting the transverse mercator projection but is failing to interpret the datum.

This is happening during export to PostGIS, and import from PostGIS. Manifold also fails to correctly import datasets from Oracle, as each time I do this I have to go 'assign projection...British National Grid'

In the past this has not been too much of a problem as we just used to import (as opposed to link) datasets from Oracle and reassign the projected coord system. We are looking at replacing Oracle with PostGIS and have been testing 'linking' of drawings. If you link a drawing, you can reassign the coord system to British Grid, but when you save your project at the end, Manifold changes them all back to transverse mercator with the WGS84 datum again which is very frustrating.

I'll report this to tech as a bug...I can't see how this can be anything else and I'm surprised no one has come up against this before.

... and it's relevant especially with PostGIS due to the constrain allowing only the one epsg written down in geometry_columns.srid to be added to the Drawing. So you can't go with any 'nearest found projection'

Let me add: I haven't analysed this to the ground because I'm absolutly happy with Manifolds Sharing DB model. perhaps a solution for you as long as you completely switch over to manifold.

Ahh.... I see Klaus beat me to a reply! :-) Here it is anyway...

if I have a drawing whose projection is set to British National Grid

You have to look at the specifics, each and every projection parameter. From your posting I see you have Manifold, CadCorp and PostGIS in play. ( It would be helpful if you told us what versions of each. ) There are three different projections mentioned: "British National Grid," "OSNI 1952 Irish National Grid," and "EPSG 27700." Note that words like "British National Grid" are often used in the case of national projection systems to refer to a particular generic coordinate system, such as Transverse Mercator, with specific parameters.

There are therefore nine different possible combinations of what those high-level projection descriptions might actually mean in terms of projection parameters. Note that Manifold (like most GIS packages) does not use EPSG codes: it does projection matching using factors such as the actual parameters said to be in use. From the Spatial DBMS Facilities topic:

Native drawings store coordinate systems using means specific to that particular spatial database. Such means usually include a table or a set of tables which store the definitions of all coordinate systems together with a table storing the bindings between columns in the database tables which contain geometry data and their coordinate systems.

Manifold can handle native drawings within DB2, Oracle and PostgreSQL and SQL Server 2008 spatial. Exporting a native drawing from Manifold allows selecting a coordinate system from the list of those registered in the database and creates the binding between the geometry column in the exported table and that coordinate system. Manifold will attempt to match the coordinate system in use to a coordinate system supported by the spatial DBMS. If an exact match is not found, Manifold will re-project the data into the closest coordinate system (projection) found. Importing or linking a native drawing into Manifold reads the binding and retrieves the coordinate system.

I'm not a PostGIS guy, so I don't know the details of how PostGIS exposes what it thinks is a list of coordinate systems it knows. However, if it advertises EPSG 27700 with coordinate system parameters expressed as a generic Transverse Mercator projection (which could be identical to something also known as "British Grid," just expressed as Transverse Mercator) or a closer match to a Transverse Mercator projection, well, that's how Manifold will match it.

So, this could be nothing, just a different name used for the same thing. Or, it could be something, where the projection parameters in use are slightly different. Only by looking at the projection parameters in use as well as those defined for each specific system can you determine that.

Dimitri....coordinate systems should not be this difficult. I've been 'doing' GIS for nearly 10 years so I'm pretty sure this is not something I'm doing wrong.

I've enclosed a text file detailing the various coordinate systems. Although I started talking about PostGIS, I've done these tests with Oracle because essentially I'm getting the same issue. Stuff that goes into Oracle as British National Grid (airy spheroid datum and transverse mercator projection) is being imported into Manifold as transverse mercator, WGS 84 datum with no mention of any national grids???

What's concerning me the most is that stuff exported from Manifold, either to PostGIS or Oracle is being imported differently.

Attachments:
projections.txt

Dimitri....coordinate systems should not be this difficult. I've been 'doing' GIS for nearly 10 years so I'm pretty sure this is not something I'm doing wrong.

I'm not suggesting it is something you are doing wrong. I'm agreeing with you that it is difficult and, in a less emphatic way, I agree that it "should" not be that difficult. I put "should" in quotes because the difficulty arises from having choices in this world, and I value very much having choices.

Having difficulty with coordinate systems is part of the landscape when interoperating between different GIS technologies. That difficulty can be reduced to some degree, but not always successfully in all cases because different systems have different approaches to projections. In some cases, those approaches invite conflicts that cannot be resolved in advance (shapefile .prj is a great example). If there was one universal standard (OK,... all you folks in the back rows of the audience who just fell down laughing have a point...) there would be no issue. But there isn't.

Some spatial DBMS packages use EPSG because Oracle does as well. But EPSG covers only part of the many projections out there, fewer than Manifold does, so you still need the superset capability. Manifold uses a more generalized interface to provide wider coverage of possible coordinate systems to provide that superset. Note that at least Manifold's system allows you to adapt, as in this case.

Would it make sense to have a dual system that would provide both EPSG direct or, in the alternative, the current superset? Sure, if enough people want that. To date, there's not been much data defined by EPSG compared to the usual way of enumerating some conventional name of a projection accompanied by a more or less complete list of parameter names and values. So you can't live on EPSG alone but you have to have a superset. If you are going to have a superset, may as well do that first and find a way of including EPSG within that.

Inevitably if you allow automatic preservation within the system as Manifold does (all the coordinate systems are always kept in shape no matter what... use Tools - Make Image to create a snapshot and that image inherits correctly the coordinate system and georegistration), you discover you have to have a way of interpreting and matching and then reprojecting on the fly into the closest match between what could be some, say, customized coordinate system and, say, EPSG. That ability to automatically match and, if need be, re-project on the fly, a very big convenience, is one of the things which necessitates the presence of an automated coordinate system discovery methodology.

Discovering the intended coordinate system by reading those values also has a social engineering dimension behind it as well, because surprisingly often projected data is accompanied by a name and coordinate system parameters that don't quite match. That's why the implied cynicism in my original post, which says you have to look at the values actually used for the coordinate system and not the names. Manifold's approach to matching projections by looking at the actual values used works out well in such cases because it at least will bring the data in accurately even if it highlights a naming issue to be resolved.

Of course, although usually Manifold gets it right with many thousands of combinations in common usage it doesn't always discover the intended system correctly. In the particular case of this thread, it looks like a difference between the conventional single precision values and Manifold's double precision values prevented a correct discovery of the datum. That's not the first time, nor will it be the last time that happens, but when it does happen in the case of a particular datum or coordinate system it can't happen very much before someone reports it and it gets fixed.

As for making coordinate system interoperability easier, that's easier to do in limited situations, such as pairwise interchange between two systems which both agree to use a particular EPSG list and not allow anything else. ...or for multiple systems which agree to use only lat/lon WGS84 within only a specific evolution of the WGS84 datum (it changes over time). But to accomplish interchange between any two GIS systems using any of the coordinate systems in use will require more difficulties than any of us would like.

Dimitri, I appreciate your comments. Having these issues is most frustrating because whilst you have two separate GIS systems writing to PostGIS or Oracle in their respective native formats, although technically you have geometry in the database which can be shared (and indeed this is the case between Cadcorp and Manifold) you end up in a situation where in one GIS system the geometry is slightly in the wrong place.

Without getting into the details of the projections again, in the projections dialogue, I fail to see any significant difference between British National Grid (from the drop down selection box) and transverse mercator, if you look at the XML data that's produced when you export a file with each of these coordinate systems specified.

I think this warrants a note to tech as I don't believe Manifold is functioning correctly with respect to database coordinate systems Manifold coordinate systems, particularly with respect to the British National Grid. If you export a drawing which was quite implicitly created in British National Grid (Transverse Mercator OSGB 1936 datum) and you bring it straight back into Manifold, why does it bring it in as Transvers Mercator (WGS84 datum)?

Something's definitely not right here.

If Manifold doesn't recognize the datum it will use the default, which is WGS84. British National Grid *is* Transverse Mercator, by the way. See http://www.ordnancesurvey.co.uk/oswebsite/gps/information/coordinatesystemsinfo/guidecontents/guidea.html for the Transverse Mercator parameters used to define the National Grid (OS leaves off the "British" part, reasonably enough taking that for granted).

I want to bring this thread up. I spend also long time with projections problem.

The most important projection in Slovakia and Czech republic is not implemented in Manifold (ESRI 102067 S-JTSK East/North - we hope it will change in version 9). So we need to set custom projection.

We have PostGIS database with correctly configured 102067 projection in spatial_ref_sys. By little reverse engineering using Manifold scripts, we found parameters for XML definition of custom projection which we are using now.

When exporting drawing to PostGIS, Manifold will detect correct projection in PostGIS and indicates 100 percent match between our custom Manifold projection and projection in PostGIS ("Projections are exactly same" will appear).

1st problem: When linking previously exported layer back to Manifold, it will not asign correct projection. I need to do it manually. Is Manifold looking also in custom projections when trying to find the best projection for linked layers ?

2nd problem: I am really sure that projection defined in PostGIS is correct. Our custom defined Manifold projection indicates 100 percent match with PostGIS projection. Still, when I reproject data to another projection, result is clearly incorrect. Where can be the problem ?

Similar?

http://209.220.176.44/forum/t82712.22#83155

Yes, problem #1 is similar to http://209.220.176.44/forum/t82712.22#83155 , has anybody submitted question to Manifold Tech and received some answer to this ?

No as I saw this post from AdamW some time ago and guessed that it was referring to the same thing

http://209.220.176.44/forum/t58538.7#58571

Thanks for this, maybe it should be my case #1. I hope version 9 will come soon.

I had a similar issue with a State Plane coordinate system, only it wasn't the datum, but a difference in scale factors between the Manifold definition and the PosGIS definition of the same projection (see post http://209.220.176.44/forum/t77143.1)

I did contact Tech Support - I reported it as a bug. I got an email back from them explaining what/why I was having the problem, and that they would have a fix for the issue in V9. I would suggest that you use the "official" method and email Tech Support on any projection issues you have, so they may be addressed in future versions..

Did You used standard support token or developer token for reporting this bug ?

I did contact Tech Support - I reported it as a bug. I got an email back from them explaining what/why I was having the problem, and that they would have a fix for the issue in V9.

I didn't recall anything like the above so I took the liberty of doing some research into this. The above is not exactly correct, so to restore full precision let me report on this as well.

The issue you experienced arose because Manifold uses more precise values for some parameters in the Manifold version of the projection than PostgreSQL uses for its version of the projection. That extra precision is enough to mark the projection as used by Manifold as "different" than the version of the projection used by PostgreSQL. It's true that if you care about the fullest possible precision, it is indeed different with PostgreSQL being very slightly less precise.

It's not Manifold's habit to require the product to be less precise in order to match the behavior of other products which are less precise, but, in this case the difference is so small that what tech support replied to you was:

We are considering augmenting our code which determines whether or not two projections are the same to ignore really small discrepancies like the one in this example, in future versions of Manifold. Until we do this, the projections will be considered different.

Two important things to note from the above:

First, note that at most tech support said they are considering adding the above new feature. They are not saying they are going to do it all, let alone in any particular future version of Manifold. They might do it, they might not. There's always the question when you allow something less precise, just where do you draw the line between cutting some slack for a less precise system and something that is so much less precise that it should be rejected as a blunder? With a system like PostgreSQL that's often hacked by people all over the world in a highly inexpert manner you don't want to just accept outright blunders even if you might be willing to accept some slightly lower precision in a "no harm done" way.

Second, it's not a "fix" but would be a new feature, an option to dumb down the product a bit to allow more convenient interchange with less precise or less rigorous systems. This falls more or less in the catagory of requests to accept broken formats and other relaxations of standards, rigor, precision, etc., which technically violate some canon but which can be useful to have. See the discussion in the Suggestions tips page.

Also...

Did You used standard support token or developer token for reporting this bug ?

No tokens are required for bug reports. If you want an answer back, you'll need a developer token for such things. See the Technical Support topic at http://www.manifold.net/doc/technical_support.htm

That tech support responded to the bug report provided by rfriedman was a courtesy. They found it interesting and so they treated it as a suggestion, which quite often will get some sort of a reply. That is not something anyone should expect without a token.

In this case, they might have been curious of his opinions on this, what he'd think if a new feature was introduced to allow projection matching to less precise projection definitions. So they let him know their view on the matter in case he wanted to reply.

There is a lot of room for such opinions. Some people will flat out say that any difference should not be accepted. Others will say it depends on how you do it. For example, should it just be some fixed epsilon where a precision difference less than that is accepted? Should a warning dialog be popped open that Manifold is relaxing its standards a tiny bit? Should the epsilon allowed be settable in Tools - Options? Should it be different for commercial software like Oracle and open source like PostgreSQL, under the theory that there's a lot greater variability in opinions expressed in the latter case?

As always, if anyone has any opinions on the above I'd encourage them to read the Suggestions tips at http://www.manifold.net/info/suggestions.shtml and to contribute their opinions.

... to allow projection matching to less precise projection definitions.

I like the idea of a settable epsilon in Tools - Options and no popup. I run into this everyday when importing stuff from ESRI as their default for State Plane is less precise. Its an unnecessary nuisance they way it stands now and any of the options you mention would be better.

As always, if anyone has any opinions on the above I'd encourage them to read the Suggestions tips at http://www.manifold.net/info/suggestions.shtml and to contribute their opinions.

Don't forget to vote! As they say in Chicago, "Vote Early and Vote Often!" :-)

Thanks to all, I will try to ask Manifold Tech.