For some reason there was an anomaly in one of our metric last night and it was ruining a lot of our graphs readability. The data was correct and we are looking into the issue but it was anomalous and so high that it was hurting mean, mean_90, etc.
So I needed to find a way to overwrite existing data in Graphite. You cannot remove datapoints in Graphite but you can overwrite the data manually if you know the time stamp:
echo 'stats.timers.browser.timedPageRender.circus.accounting.rendered.mean 14000 1409268080' | nc p2-graphite.ad.prodcc.net 2003
If you’ve got a few bad data-points and you used statsd to gather timer data for example and as such it spans accross a few different metrics (median, std, mean, mean_90, …), I wrote a quick commmand line application to overwrite data over a give threshold for a span of time. You’ll find it in this gist but proceed with caution. Look at and understand the different constants, check the time range (I didn’t put constants for that). Here there be dragons. You’ve been warned!