We talk about the energy problem being very complex. Just the other day, a clean web hacker was telling me why many coders miss the mark in disrupting energy: they do NOT understand the complexities of the industry.
JD Hammerly’s note on Why you should fire your coders (and hire solvers instead) seems to be the vision of SURGE, it takes something complex and simplifies it down into individual parts that can be digested and nailed. But is “it” that simple? Can the BIG CODERS from energy who boast about how many lines they have written be able to pivot their mindset?
A group of incredibly smart entrepreneurs with backgrounds in part of the energy equation are working on bringing the CleanWeb Hackathon TEXAS style down to Houston. If JD is correct, the CleanWeb Hackathon, this time in Boston, is a great entrepreneurial way to disrupt organizations that cannot easily fire their coders and pivot into a new way of thinking about these problems.
Hackers Unite…If interested in Texas, keep posted.
Do you agree with JD?
3 comments
I’m of two minds here.
On one hand, he’s preaching to the choir! I’m a firm believer in not re-inventing the wheel, particularly with highly mathematical code. When designing a system with a heavy computational or analytics requirements, one should make use of existing, mature, tested, and supported computational tools as much as possible. Make sure that the surrounding infrastructure is designed to connect to these mature tools in a seamless way. If your analytics team has to rewrite existing, refined algorithms in another language in order to connect them to your data sources, you’ve probably made a bad architecture choice.
On the other hand, I would argue that he’s simplifying things for his audience. I have two specific examples from the text, and one caveat:
Example 1
Emphasis mine: “To use them effectively, developers *merely* need to know the basics of data, how to present the problem to the solver, and which of several different mathematical approaches to use (or, at least, how to test different approaches to see which one gives the best results).”
This is like saying “to beat Tiger Woods one *merely* needs to complete the course in fewer strokes.” What Mr. Hammerly describes here is the core challenge of a solver-driven approach. This is not something you trust some random coder to do. You need people with experience using these tools and at least a genuine understanding of the mathematical concepts they implement.
For instance, you don’t need to know how to code up your own interior-point or dual simplex solver to effectively use linear programming; and that’s a good thing. But if you don’t know what that last sentence means, you’re not the one that should be trusted to integrate an LP solver into your application, or to even know if it’s going to be useful in the first place.
Example 2:
“You can switch from one mathematical approach to another without investing in any code.” This, I’m afraid, is BS. It’s highly, highly unlikely that your domain-specific problem will present itself in a form that is a perfect match for any commercial solver. There will inevitably be some code that must be written to translate from your domain to the standard form. This code may seem trivial to an analytics expert, but it wouldn’t be trivial to someone who doesn’t have a firm grasp of the algorithms.
Caveat
There are going to be situations where a custom solver is necessary. One of the problems with off-the-shelf solvers is that they sacrifice performance for generality. Yes, many have been tuned to the hilt; they’ll be as fast as they possibly can be. But sometimes your mathematical models will have a very specific, unique structure and there just isn’t any way for the solver as written to take advantage of it. Sometimes building a custom solver that exploits that unique structure can bring huge performance boosts. (I’ve personally seen 1000x or more in extreme cases.)
What do you do in this case? First, you use an off-the-shelf solver to get your numbers right; then you re-implement specific pieces to get the performance you need. The two stage approach is important: first, it puts of the difficult coding until you’re sure you need it (what if you decide to go with a completely different solver?). Second, your functioning (but slow) implementation is a great baseline against which to test your high-performance code.
I’m sure I’m oversimplifying things here, too. Overall, I think Mr. Hammerly’s advice is good. Reliance on existing solvers allows companies to deploy much more powerful analytical algorithms than they could ever consider using if they had to implement them all on their own. But using commerical solvers does not eliminate the need for true analytics experts on your team. It just means that you can use fewer, and they will be a lot more productive.
[…] Hacking in Energy, Made Easy? (garagesalemba.com) […]
You may need to invest by getting 1 to 5 people per country to
aggregate the schedule and content. If a hacker notices a certain password used for
multiple accounts, he or she will use it for one’s other accounts in order to steal
valuable information. I went over to him and without saying anything pointed my mobile phone camera in his face and took
a picture.
Comments are closed.