ImportFromFME Published


#1

ImportFromFME is a package that contains nodes allowing you to import data into Dynamo through Safe Software’s FME Spatial ETL tool.

FME has readers for over 400 CAD, GIS, Spatial Database, BIM and online formats. For more information, please visit www.safe.com.

This is a early version of this package, and we welcome all suggestions for improvements.

The nodes are presently Python based, using FME’s .NET API. They should serve as good examples for any who wish to do their own FME integrations into Dynamo.

There are two nodes in this package:

ImportTableFromFME - imports tables from any supported FME format (ie. Excel, Oracle, SQL Server, Geodatabase) as a list of rows, similar to the Excel or CSV reader nodes.

ImportGeometryFromFME - imports geometry from most supported FME formats, with the output separated by geometry type. If the source data has coordinate system information, it can be automatically reprojected to the local Revit coordinate system, based on Project Location and Units.

Requirements

Dynamo Revit, version 1.3.2 or newer
FME 2018.0 or newer must be installed and licensed

Installation

The package can be installed from the Package Manager

Usage

ImportTableFromFME has three inputs:

FME_Home_Folder: This is the location of the root FME installation folder. This is needed in order to connect to the FMEObjects .NET dll. Please use a String node to supply the location.

FME_Connection_String: This is a very long string that provides FME Objects with the reader information, including format, dataset, coordinate system, table to read etc. You can create this string using the utility FMEPCConfg.exe, which can be found in the FME installation folder. Run the utility, then create a Reader connection to the dataset. Copy the Connection Parameters string from the Connection text box and paste it into a String node in your graph. The utility should be run on the same computer as Revit, to ensure that FMEObjects will have the required database connections.

FMEPCConfig2

FME_Logfile: This is an optional input that provides the location of the FME log file created by the import. This can be useful when an import fails. This input should be a String as well.

ImportFMEGeometry uses the same inputs as above, as well as a Reproject switch.

It should be able to import most types of geometry into Dynamo. At present, it will ignore rasters, point clouds and text. If the Reproject switch is set to True, and the source data has a coordinate system, the node will get the Project Location, Rotation and Unit from the Revit file and use that to reproject the source data to match the Revit data.

The node has a separate output for each geometry type to make it easier to choose the geometry you want to use. It also has a separate output for the Donut Hole polygons, since Donuts are not supported in Dynamo (as far as I could tell, anyway).

There is also an output for any features that creates a Dynamo geometry creation error. This output includes the Dynamo error, feature geometry type, FME feature type, and list of the coordinate points.

Example

Importing Pipes and Structures from Civil-3D file, in Illinois State Plane coordinate system.

Civil-3D

Dynamo


#2

#3

Very nice, will be trying it soon! So to be clear, this does not require you to author anything in FME because the connection string generator utility builds the connection string based on the data you’re reading and sending back to Dynamo, correct?


#4

Hi David,
Yes, the node is just using the FME reader, not any of the Workbench processing. If you want to pre-process the data with FME, you can turn your workspace into a Custom Reader, and use this as the data source.


#5

Hi, I’ve tried this, but can’t get it to work… I’ve run FMEPCConfig.exe file and gotten the connection string, but I can’t seem to find the FMEObjectsDotNet4.dll in the python script within the ImportTable node.

Any idea, @dcampanas?


#6

That is strange - you seem to have the correct location set.

My first guess would be a bit depth mismatch between Dynamo and FME, but it looks like you have the 64 bit version installed.

Are you able to view the same table in the FME Data Inspector? This will tell us if FME is working correctly.


#7

Hi, yep. It’s 100% readable in the Data Inspector.

If I however run the Dynamo script with FME open, it produces a new error message: image


#8

I can reproduce the second error - it happens when setting the log file location to a non-existent folder. Please try setting the log location to a folder that already exists, or creating the folder C:\Desktop.

Once you change the location, would you mind sending me the resulting log file? It would help me diagnose the problem loading the DLL. You can send direct to dave.campanas@safe.com, if you prefer.


#9

late response here… I used a file path node with an existing folder and added the logfile name and it works, thanks! :slight_smile:
image


#10

Hi Dave,

Thanks for sharing this package.

I have been trying to use the FEM API to write back to a SDF3 database using code based on your read code but have not been having any luck.

Simply, what I am trying to do is to use python to read a SDF3 database, as the reader steps through the features I would like to update some attributes and write it back to the database. I am trying to do this using the code below but have not been having any luck. It seems to run in dynamo however it is not updating the source database. Please note that I have not worked with databases much before so there may be a fundamental error in the way that I am approaching this.

Any advice would be much appreciated.

import sys
import clr
clr.AddReference('ProtoGeometry')

from Autodesk.DesignScript.Geometry import *
from System.Collections.Specialized import *
from System import Array

#The inputs to this node will be stored as a list in the IN variables.
fmeHome = IN[0]
connectionString = IN[1]

# import FMEObjects .NET API from FME Home
sys.path.append(fmeHome)
clr.AddReferenceToFile('FMEObjectsDotNet4.dll')
from Safe.FMEObjects import *

# replace outer commas in connection string with semi-colons to simplify parsing
qflag = 0
parsedString = ''
for c in connectionString:
    if c == '"':
        qflag = abs(qflag - 1)
    else:
        if c == ',' and qflag == 0:
            c = ';'
        parsedString = parsedString + c

# parse connection string to get parameters
connectionList = parsedString.Split(';')
sourceFormat = connectionList[1]
sourceDataset = connectionList[3]

# convert connection list to Stringcollection for reader
rparams = StringCollection()
for param in connectionList:
    rparams.Add(param)

# create FMEObjects session
nullParams = StringCollection()
session = FMEObjects.CreateSession()
session.Init(nullParams)

# create blank output list
outlist = []
# create log file for debugging
if IN[2] != None:
    log = session.LogFile()
    log.SetFileName(IN[2], False)

# open new reader and writer using connection string
reader = session.CreateReader(sourceFormat, False, rparams)
reader.Open(sourceDataset, nullParams)

writer = session.CreateWriter(sourceFormat, rparams)
writer.Open(sourceDataset, nullParams)

# read first table
feature = session.CreateFeature()
moreSchemas = reader.ReadSchema(feature)
tableName = feature.FeatureType

# creates feature vector and adds the datasets first feature
featureVector = session.CreateFeatureVector()
featureVector.Append(feature)

# get names of attributes in table
attrs = StringCollection()
feature.GetAllAttributeNames(attrs)

        
# convert collection of attribute names to list and keep only user attributes
attrList = []
titleList = []
for i in range(attrs.Count):
    attr = attrs.Item[i]
    if attr.startswith('*') or attr.startswith('fme_'):
        pass
    else:
        attrList.append(attr)
        titleList.append(attr)

titleList.append('Geometry')  
# set attribute names as first item in table list
outlist.append(titleList)

# loop through features in table
while reader.Read(feature):
    featureList = []
    # get all attribute values from feature
    if feature.FeatureType == tableName:
        featureVector.Append(feature)
        writer.GetSchemaFeatures(featureVector)

        feature.SetStringAttribute("Number of Storeys", "15")
        writer.StartTransaction()
        writer.Write(feature)
        writer.CommitTransaction()
        


        """for attr in attrList:
            value = feature.GetStringAttribute(attr)
            featureList.append(value)
        featureList.append(feature.ExportGeometryToOGCWKT())
        # append feature attribute values to table list
        outlist.append(featureList)"""
writer.Close()



#Assign your output to the OUT variable.
OUT = outlist

#11

Hi Harry,

I am happy to see that you are exploring FMEObjects within Dynamo. I admit that the mix of Python and .NET can make things challenging.

I think a good start would be to move the:

writer.GetSchemaFeatures(featureVector)

line out of the reader.Read loop. The schema only needs to be set once for each output table.

Generally, for a simple read/write, we would have two loops:

  • the first loop reads the reader schema features, then writes them to the writer.
feature = session.CreateFeature()
while reader.ReadSchema(feature):
    writer.AddSchema(feature)
    feature = session.CreateFeature()
  • the second loop reads the features and writes them to the writer.
feature = session.CreateFeature()
while reader.Read(feature):
    feature.SetStringAttribute("Number of Storeys", "15")
    writer.write(feature)
    feature = session.CreateFeature()

All the FMEObjects APIs are wrappers around C++, which passes the feature object to the vector and writer, so we need to create a new feature for each read operation.

If you installed FME with the SDK, you can find sample applications in FME\fmeobjects\samples that may be helpful.

Kind regards,

Dave Campanas


#12

Hi Dave,

Thanks for your prompt reply.

I have attempted to update my script as outlined above but the writer still is not updating the database.

I tried a couple of different ways to feed the updated features into the writer, they are shown below:

  • The way you suggested. To make this work I had to recreate the reader so we could cycle through the features again.

      feature = session.CreateFeature()
      reader = session.CreateReader(sourceFormat, False, rparams)
      reader.Open(sourceDataset, nullParams)
      while reader.Read(feature):
     	    feature.SetStringAttribute("Number of Storeys", "15")
     	    writer.Write(feature)
     	    feature = session.CreateFeature()
    
  • I have also tried by extracting the features back out of the feature vector with the following code (probably not the best implementation of accessing the features from the feature vector)

      feature = session.CreateFeature()
      featureList = [] #testing to see if features are being accessed
      moreFeatures = True
      while moreFeatures == True:
         try:
             feature = featureVector.RemoveLast()
             featureList.append(feature)
             feature.SetStringAttribute("Number of Storeys", "15")
             writer.StartTransaction()
             writer.Write(feature)
             writer.CommitTransaction()
          except:
              moreFeatures = False
    

I have tried the both methods with and without the use of transactions and both will run without throwing an error, however the database file will not be updated. I had also set up the schema for the output table prior to the loops above using writer.GetSchemaFeatures(featureVector)

I am unsure if I am setting up the writer correctly. As I am trying to write back to the database I am reading from I have been using the data that was used to set up the reader.

# parse connection string to get parameters
connectionList = parsedString.Split(';')
sourceFormat = connectionList[1]
sourceDataset = connectionList[3]

# convert connection list to Stringcollection for reader
rparams = StringCollection()
for param in connectionList:
rparams.Add(param)

# create FMEObjects session
nullParams = StringCollection()
session = FMEObjects.CreateSession()
session.Init(nullParams)

# create blank output list
outlist = []

# create log file for debugging
if IN[2] != None:
log = session.LogFile()
log.SetFileName(IN[2], False)

# open new reader and writer using connection string
reader = session.CreateReader(sourceFormat, False, rparams)
reader.Open(sourceDataset, nullParams)

writer = session.CreateWriter(sourceFormat, rparams)
writer.Open(sourceDataset, nullParams)

I had a quick look at the samples in the folder directory you showed above but the only writing example that I found was using the FMEUniversalWriter().

Kind regards,

Harry


#13

Hi Harry,

Sorry, I should have caught this previously. FME’s SDF3 writer can write to the same SDF file as you are reading, but only if the writer parameter OVERWRITE_FILE is set to No, and you are writing to a different table.

I think the safest thing might be to write to a new SDF database. This way you don’t need to change the table name. Alternatively, you could read from a copy of your SDF, then write to the original, with Overwrite Table set to Yes.

Kind regards,
Dave Campanas


#14

Hi Dave,

Thanks getting back to me again, I reworked the code to do as you advised and I have still not had any luck getting the writer to work. I also tried generating a writer connection string using the FMEPCConfig.exe and updated the overwrite parameters.

At this stage we are looking a moving away from using an SDF database to something more universally accessible, so it will not be to much of a problem if I cannot get the writer to work. I have included my updated code and the writer connection string if you wanted to have a look through it. If not, no worries and thanks for your help so far :slight_smile: .

Writer Connection String:
SOURCE_FORMAT,SDF3,SOURCE_DATASET,"E:\Current Jobs\00 Not Prejects\Dynamo and Navis Training\Projects\Automation - Solution Architecture Document\DSSTest - Copy.sdf",RUNTIME_MACROS,"OVERWRITE_FILE,Yes,DEFAULT_SCHEMA_NAME,Default,TABLE_DEFAULTS,,OverwriteTable,YES,WRITER_MODE,INSERT,IdentityColumnName,FeatId,GeometryColumnName,Geometry,ADVANCED,,XY_TOLERANCE,0.0,Z_TOLERANCE,0.0,DESTINATION_DATASETTYPE_VALIDATION,Yes",META_MACROS,"DestOVERWRITE_FILE,Yes,DestDEFAULT_SCHEMA_NAME,Default,DestTABLE_DEFAULTS,,DestOverwriteTable,YES,DestWRITER_MODE,OVERWRITE,DestIdentityColumnName,FeatId,DestGeometryColumnName,Geometry,DestADVANCED,,DestXY_TOLERANCE,0.0,DestZ_TOLERANCE,0.0,DestDESTINATION_DATASETTYPE_VALIDATION,Yes",METAFILE,SDF3,COORDSYS,

Script:
(I tried this with and without creating a sepperate session for writing)

import sys
import clr
clr.AddReference('ProtoGeometry')

from Autodesk.DesignScript.Geometry import *
from System.Collections.Specialized import *
from System import Array

#The inputs to this node will be stored as a list in the IN variables.
fmeHome = IN[0]
connectionString = IN[1]
writeConnectionString = IN[3]

# import FMEObjects .NET API from FME Home
sys.path.append(fmeHome)
clr.AddReferenceToFile('FMEObjectsDotNet4.dll')
from Safe.FMEObjects import *

# replace outer commas in connection string with semi-colons to simplify parsing for reader
qflag = 0
parsedString = ''
for c in connectionString:
    if c == '"':
        qflag = abs(qflag - 1)
    else:
        if c == ',' and qflag == 0:
            c = ';'
        parsedString = parsedString + c

# parse connection string to get parameters for reader
connectionList = parsedString.Split(';')
sourceFormat = connectionList[1]
sourceDataset = connectionList[3]

# replace outer commas in connection string with semi-colons to simplify parsing for writer
qflag = 0
parsedString = ''
for c in writeConnectionString:
    if c == '"':
        qflag = abs(qflag - 1)
    else:
        if c == ',' and qflag == 0:
            c = ';'
        parsedString = parsedString + c

# parse connection string to get parameters for writer
writeConnectionList = parsedString.Split(';')
writeSourceFormat = writeConnectionList[1]
writeSourceDataset = writeConnectionList[3]


# convert connection list to Stringcollection for reader
rparams = StringCollection()
for param in connectionList:
    rparams.Add(param)

# convert connection list to Stringcollection for writer
writeParams = StringCollection()
for param in writeConnectionList:
    writeParams.Add(param)

# create FMEObjects session
nullParams = StringCollection()
session = FMEObjects.CreateSession()
session.Init(nullParams)

writeSession = FMEObjects.CreateSession()
writeSession.Init(None)

# create blank output list
outlist = []
# create log file for debugging
if IN[2] != None:
    log = session.LogFile()
    log.SetFileName(IN[2], False)

if IN[2] != None:
    logWrite = writeSession.LogFile()
    logWrite.SetFileName(IN[4], False)

# open new reader and writer using connection string
reader = session.CreateReader(sourceFormat, False, rparams)
reader.Open(sourceDataset, nullParams)

writer = writeSession.CreateWriter(writeSourceFormat, None)
writer.Open(writeSourceDataset, writeParams)


# read first table
feature = session.CreateFeature()
moreSchemas = reader.ReadSchema(feature)
tableName = feature.FeatureType

# creates feature vector and adds the datasets first feature
featureVector = session.CreateFeatureVector()
# featureVector.Append(feature)

# get names of attributes in table
attrs = StringCollection()
feature.GetAllAttributeNames(attrs)

          
# convert collection of attribute names to list and keep only user attributes
attrList = []
titleList = []
for i in range(attrs.Count):
    attr = attrs.Item[i]
    if attr.startswith('*') or attr.startswith('fme_'):
        pass
    else:
        attrList.append(attr)
        titleList.append(attr)

titleList.append('Geometry')  
# set attribute names as first item in table list
outlist.append(titleList)

# loop through features in table
while reader.Read(feature):
    featureList = []
    # adds features to feature vector
    featureVector.Append(feature)
    # get all attribute values from feature
    for attr in attrList:
        value = feature.GetStringAttribute(attr)
        featureList.append(value)
    featureList.append(feature.ExportGeometryToOGCWKT())
    # append feature attribute values to table list
    outlist.append(featureList)
    feature = session.CreateFeature()

# sets writer schema
writer.GetSchemaFeatures(featureVector)

feature = writeSession.CreateFeature()

writeList = [] #test to see what is being updated
featureList = [] # test to see what features are in feature vector

# loops through feature in feature vector updating a value and writing it to the database
# This is an alternate loop to what is used below
moreFeatures = True
while moreFeatures == True:
    try:
        feature = featureVector.RemoveLast()
        featureList.append(feature) # testing what feature is being updated
        writeList.append(feature.GetStringAttribute("Number of Storeys")) # testing value of feature being updated
        feature.SetStringAttribute("Number of Storeys", "15")
        writer.StartTransaction()
        logWrite.LogFeature(feature, 0, -1)
        writer.Write(feature)
        writer.CommitTransaction()
        #feature = session.CreateFeature()
    except:
        moreFeatures = False
    

"""# rereading source features from start of list
reader = session.CreateReader(sourceFormat, False, rparams)
reader.Open(sourceDataset, nullParams)
# loops through features writing updated features to database
while reader.Read(feature):
    feature.SetStringAttribute("Number of Storeys", "15")
    writer.StartTransaction()
    writeList.append(feature.FeatureType) # test to see what table is being writen too
    logWrite.LogFeature(feature, 0, -1)
    writer.Write(feature)
    writer.CommitTransaction()
    feature = writeSession.CreateFeature()"""


#Assign your output to the OUT variable.
OUT = outlist

Thanks again for your help.

Cheers,

Harry


#15

Hi Harry,

I made a few major changes to your code:

  • I stripped out all the code to create the Dynamo output list, for clarity
  • I removed the extra session - there can be only one active FME session
  • I replaced the writer.GetSchemaFeatures method (which is used for getting the writer schema) with writer.AddSchema

Sorry about not catching the GetSchemaFeatures method previously - I really should have examined it more closely. I am more familiar with the Python API, and thought this was something different in our .NET API. I have updated my previous post with a fixed example so as not confuse other readers of this stream.

Please try the following:

import sys
import clr
clr.AddReference('ProtoGeometry')

from Autodesk.DesignScript.Geometry import *
from System.Collections.Specialized import *
from System import Array

#The inputs to this node will be stored as a list in the IN variables.
fmeHome = IN[0]
connectionString = IN[1]
writeConnectionString = IN[3]

# import FMEObjects .NET API from FME Home
sys.path.append(fmeHome)
clr.AddReferenceToFile('FMEObjectsDotNet4.dll')
from Safe.FMEObjects import *

# replace outer commas in connection string with semi-colons to simplify parsing for reader
qflag = 0
parsedString = ''
for c in connectionString:
    if c == '"':
        qflag = abs(qflag - 1)
    else:
        if c == ',' and qflag == 0:
            c = ';'
        parsedString = parsedString + c

# parse connection string to get parameters for reader
connectionList = parsedString.Split(';')
sourceFormat = connectionList[1]
sourceDataset = connectionList[3]

# replace outer commas in connection string with semi-colons to simplify parsing for writer
qflag = 0
parsedString = ''
for c in writeConnectionString:
    if c == '"':
        qflag = abs(qflag - 1)
    else:
        if c == ',' and qflag == 0:
            c = ';'
        parsedString = parsedString + c

# parse connection string to get parameters for writer
writeConnectionList = parsedString.Split(';')
writeSourceFormat = writeConnectionList[1]
writeSourceDataset = writeConnectionList[3]

# convert connection list to Stringcollection for reader
rparams = StringCollection()
for param in connectionList:
    rparams.Add(param)

# convert connection list to Stringcollection for writer
writeParams = StringCollection()
for param in writeConnectionList:
    writeParams.Add(param)

# create FMEObjects session
# only need one session object
nullParams = StringCollection()
session = FMEObjects.CreateSession()
session.Init(nullParams)

# create blank output list
outlist = []

# create log file for debugging
if IN[2] != None:
    log = session.LogFile()
    log.SetFileName(IN[2], False)

# open new reader and writer using connection strings
reader = session.CreateReader(sourceFormat, False, rparams)
reader.Open(sourceDataset, nullParams)

writer = session.CreateWriter(writeSourceFormat, None)
writer.Open(writeSourceDataset, writeParams)

# read table definitions
feature = session.CreateFeature()
while reader.ReadSchema(feature):
    # adds table definition to writer - don't use GetSchemaFeatures, 
    # which is for getting writer schemas, not setting
    writer.AddSchema(feature)
    # create new feature, since other was consumed by writer
    feature = session.CreateFeature()

# read and write all features
# features will be sent to the appropriate tables automatically
# to limit tables read/written, use reader settings in FMEPCConfig.exe
feature = session.CreateFeature()
while reader.Read(feature):
    writer.StartTransaction()
    writer.Write(feature)
    writer.CommitTransaction()
    # create new feature, since other was consumed by writer
    feature = session.CreateFeature()

#Assign your output to the OUT variable.
OUT = outlist

#16

Thanks Dave,

This code solves the problem and I am now able to write to the SDF database. Thank you for putting it together.

I am now experiencing a different issue which I am not to sure about. When I write to the SDF database for the first time everything works as expected, when I write to it a second time the script runs however the session is not closed. This leave the SDF database at about 1/4 of its file size and as it still seems to be open it is not possible to write to it again. To close the session I have to close Revit completely. When I do close Revit the database returns to it full size and the following line is added to the log file:

2018-09-12 11:33:41| 86.2| 14.8|WARN |Warning: not all FMESessions that were created were destroyed before shutdown. This may cause instability

The only other differences that I have noticed between the log files on the first and second run are that the SDF3 module will not be reloaded and the FDO connection is not re-established.

I have closed and disposed of the reader, writer and session at the end of the script however it does not seem affect the outcome.

writer.Close()
writer.Dispose()
reader.Close()
reader.Dispose()
feature.Dispose()
session.Dispose()

OUT = outlist

Is there any reason the Dispose() method would not be successful or is this a Dynamo / Revit issue?

Kind regards,

Harry