Peter Coles

Author: Peter

Big Data Seeding Part I – Importing Large Datasets into Laravel

I love Laravel’s data seeding. I use it to create test and sometimes initial production data on every Laravel project that I do. But recently I was presented with a (very) large lump of pre-existing test data.

A bit of background … I was extracting microservices from a monolithic legacy system, and had decided that to minimise risk, I’d maintain data compatibility between old and new systems. With Laravel, this was really easy as Laravel models allow us to define table names and relationships even when they don’t conform to Laravel’s default expectations.

Having an existing, large scale, integration test database looked like a huge benefit, and it was. It would have taken ages to construct this much data from scratch, but thought of translating it to into seeding classes filled me with dread. Happily there’s a much better way of handling this.

Continue reading

I Think I Worked on This System

<shaking head> I think I worked on this system </shaking head>

(credit xkcd)

Using Elixir outside of Laravel

I recently inherited a medium-sized Symfony application. The backend code was beautifully organized. The frontend … not so much.

I’m a big fan of Gulp and more recently have converted to Elixir, so that begged the question, could I use Elixir outside of Laravel to help me get the frontend assets under control? Happily the answer was yes. Let me show you how.

Continue reading

Bi-weekly (fortnightly) Scheduling

We have a lot of tasks run to a bewildering schedule across multiple servers. We use cron, of course. But maintaining the schedule is painful. Very painful.

We’ve tried committing copies of the crontab to a repository, but they’re almost never up-to-date.

And worse, if I need to do something even moderately complex like, say, schedule a job to run every two weeks, cron starts to get complicated.

But things just got better.

Continue reading

Command line PHP

I know that there will be people who read this and say, “well of course, you can do that”. But I also know that there others who, like me, worked with PHP for many years unaware that you could execute PHP directly from the command line without the need for a script.

But why would you want to?

Continue reading

Transferring large files

Sometimes I have to move large files between servers. Most often they’re database backups, and since we have single database tables that can extend to many hundreds of gigabytes, whole database dumps can easily generate files of one or more terrabytes.

Even more challenging was a recent need to re-construct historical data from backup drives on a development workstation and then upload the results.

But here’s the wrinkle. Once file sizes move beyond 3-4GB, transfers seem to become a smidgen unreliable, whatever protocol I use.

The answer … a very useful *nix utility called split.

Continue reading

Copyright © 2017 Peter Coles

Theme by Anders NorenUp ↑