Add notes for 2016-12-13

content/post/2016-12.md

@@ -479,3 +479,37 @@ UPDATE 35
 ```
 
 - Work on article for KM4Dev journal
+
+## 2016-12-13
+
+- Checking in on CGSpace postgres stats again, looks like the `shared_buffers` change from a few days ago really made a big impact:
+
+![postgres_bgwriter-week](2016/12/postgres_bgwriter-week-2016-12-13.png)
+![postgres_connections_ALL-week](2016/12/postgres_connections_ALL-week-2016-12-13.png)
+
+- Looking at logs, it seems we need to evaluate which logs we keep and for how long
+- Basically the only ones we *need* are `dspace.log` because those are used for legacy statistics (need to keep for 1 month)
+- Other logs will be an issue because they don't have date stamps
+- I will add date stamps to the logs we're storing from the tomcat7 user's cron jobs at least, using: `$(date --iso-8601)`
+- Would probably be better to make custom logrotate files for them in the future
+- Clean up some unneeded log files from 2014 (they weren't large, just don't need them)
+- So basically, new cron jobs for logs should look something like this:
+- Find any file named `*.log*` that isn't `dspace.log*`, isn't already zipped, and is older than one day, and zip it:
+
+```
+# find /home/dspacetest.cgiar.org/log -regextype posix-extended -iregex ".*\.log.*" ! -iregex ".*dspace\.log.*" ! -iregex ".*\.(gz|lrz|lzo|xz)" ! -newermt "Yesterday" -exec schedtool -B -e ionice -c2 -n7 xz {} \;
+```
+
+- Since there is `xzgrep` and `xzless` we can actually just zip them after one day, why not?!
+- We can keep the zipped ones for two weeks just in case we need to look for errors, etc, and delete them after that
+- I use `schedtool -B` and `ionice -c2 -n7` to set the CPU scheduling to `SCHED_BATCH` and the IO to best effort which should, in theory, impact important system processes like Tomcat and PostgreSQL less
+- When the tasks are running you can see that the policies do apply:
+
+```
+$ schedtool $(ps aux | grep "xz /home" | grep -v grep | awk '{print $2}') && ionice -p $(ps aux | grep "xz /home" | grep -v grep | awk '{print $2}')
+PID 17049: PRIO   0, POLICY B: SCHED_BATCH   , NICE   0, AFFINITY 0xf
+best-effort: prio 7
+```
+
+- All in all this should free up a few gigs (we were at 9.3GB free when I started)
+- Next thing to look at is whether we need Tomcat's access logs