Pantheon Community

Auto Importing Raw Data in Wordpress

Hello!

We have a request is to automate a process in which XML is placed on the server, read, and imported into Wordpress to then be queried and displayed in multiple locations on the site. We’ve been looking at the WP All Import plugin, which we’re familiar with as a Wordpress tool, but are unsure if it’s the best option for raw data. One of the feeds would be updated daily, and another feed would consist of a lot of historical data, updated monthly or quarterly, that then adds items/data rows as new data is required.

Again, the tricky part to us is the data being stored/visible in the Wordpress admin. There’s potentially a lot of data points, and with the historical updates we’re trying to determine the best way to organize it in a way that parts of it it can be pulled in a number of different areas on the site.

We’re wondering if our best route is:

Create separate tables in the database that only handle this raw data, outside of the general Wordpress structure, and query the data from those tables? Do we have that ability on Pantheon or is it a bad idea altogether?

Or, use the Import plugin, schedule a cron job to import the data into custom fields in Wordpress. If that’s our best option, do you have any recommendations with how to organize it? One large post with all of the necessary fields? Or split into multiple posts?

Again, just curious if someone has encountered this type of need before, and if there’s any recommendations based on your expertise as we’re exploring our options.

2 Likes

From what I am understanding of the business need, I highly recommend making a custom plugin to handle this. If it were me the custom plugin would:

  • Create the database tables (or you could possibly leverage the metadata table) to store the ingest from the XML files when the plugin is enabled.
  • Create any post types needed to display the above data when the plugin is enabled.
  • Hook into the WP cron and execute an ingest when new files appear in an upload directory
  • Potentially provide a configuration page were people could upload XML files

Next, I would solidify the process of how XML files arrive on your production server at pantheon. Either set up a file within your plugin directory, or someplace accessible to the internet. Then, set up an automated SFTP or rsync process to get the XML files there.

In short, I would recommend keeping this ingest dataset separate from WordPress’s infrastructure, so that it is easier to migrate/export if the need arises down the road.

I would also read up on Pantheon’s available file paths:


My 2 cents.