diff --git a/content/post/2018-02.md b/content/post/2018-02.md index 7f7014345..d64dd1990 100644 --- a/content/post/2018-02.md +++ b/content/post/2018-02.md @@ -951,3 +951,102 @@ SELECT * FROM pg_locks pl LEFT JOIN pg_stat_activity psa ON pl.pid = psa.pid; - Finally finished the [orcid-authority-to-item.py](https://gist.github.com/alanorth/6d7489b50f06a6a1f04ae1c8b899cb6e) script! - It successfully mapped 2600 ORCID identifiers to items in my tests - I will run it on DSpace Test + +## 2018-02-28 + +- CGSpace crashed today, the first HTTP 499 in nginx's access.log was around 09:12 +- There's nothing interesting going on in nginx's logs around that time: + +``` +# zcat --force /var/log/nginx/*.log /var/log/nginx/*.log.1 | grep -E "28/Feb/2018:09:" | awk '{print $1}' | sort | uniq -c | sort -n | tail -n 10 + 65 197.210.168.174 + 74 213.55.99.121 + 74 66.249.66.90 + 86 41.204.190.40 + 102 130.225.98.207 + 108 192.0.89.192 + 112 157.55.39.218 + 129 207.46.13.21 + 131 207.46.13.115 + 135 207.46.13.101 +``` + +- Looking in dspace.log-2018-02-28 I see this, though: + +``` +2018-02-28 09:19:29,692 ERROR org.dspace.app.xmlui.cocoon.DSpaceCocoonServletFilter @ Serious Error Occurred Processing Request! +org.springframework.web.util.NestedServletException: Handler processing failed; nested exception is java.lang.OutOfMemoryError: Java heap space +``` + +- Memory issues seem to be common this month: + +``` +$ grep -c 'nested exception is java.lang.OutOfMemoryError: Java heap space' dspace.log.2018-02-* +dspace.log.2018-02-01:0 +dspace.log.2018-02-02:0 +dspace.log.2018-02-03:0 +dspace.log.2018-02-04:0 +dspace.log.2018-02-05:0 +dspace.log.2018-02-06:0 +dspace.log.2018-02-07:0 +dspace.log.2018-02-08:0 +dspace.log.2018-02-09:0 +dspace.log.2018-02-10:0 +dspace.log.2018-02-11:0 +dspace.log.2018-02-12:0 +dspace.log.2018-02-13:0 +dspace.log.2018-02-14:0 +dspace.log.2018-02-15:10 +dspace.log.2018-02-16:0 +dspace.log.2018-02-17:0 +dspace.log.2018-02-18:0 +dspace.log.2018-02-19:0 +dspace.log.2018-02-20:0 +dspace.log.2018-02-21:0 +dspace.log.2018-02-22:0 +dspace.log.2018-02-23:0 +dspace.log.2018-02-24:0 +dspace.log.2018-02-25:0 +dspace.log.2018-02-26:0 +dspace.log.2018-02-27:6 +dspace.log.2018-02-28:1 +``` + +- Top ten users by session during the first twenty minutes of 9AM: + +``` +$ grep -E '2018-02-28 09:(0|1)' dspace.log.2018-02-28 | grep -o -E 'session_id=[A-Z0-9]{32}' | sort -n | uniq -c | sort -n | tail -n 10 + 18 session_id=F2DFF64D3D707CD66AE3A873CEC80C49 + 19 session_id=92E61C64A79F0812BE62A3882DA8F4BA + 21 session_id=57417F5CB2F9E3871E609CEEBF4E001F + 25 session_id=C3CD265AB7AA51A49606C57C069A902A + 26 session_id=E395549F081BA3D7A80F174AE6528750 + 26 session_id=FEE38CF9760E787754E4480069F11CEC + 33 session_id=C45C2359AE5CD115FABE997179E35257 + 38 session_id=1E9834E918A550C5CD480076BC1B73A4 + 40 session_id=8100883DAD00666A655AE8EC571C95AE + 66 session_id=01D9932D6E85E90C2BA9FF5563A76D03 +``` + +- According to the log 01D9932D6E85E90C2BA9FF5563A76D03 is an ILRI editor, doing lots of updating and editing of items +- 8100883DAD00666A655AE8EC571C95AE is some Indian IP address +- 1E9834E918A550C5CD480076BC1B73A4 looks to be a session shared by the bots +- So maybe it was due to the editor's uploading of files, perhaps something that was too big or? +- I think I'll increase the JVM heap size on CGSpace from 6144m to 8192m because I'm sick of this random crashing shit and the server has memory and I'd rather eliminate this so I can get back to solving PostgreSQL issues and doing other real work +- Run the few corrections from earlier this month for sponsor on CGSpace: + +``` +cgspace=# update metadatavalue set text_value='United States Agency for International Development' where resource_type_id=2 and metadata_field_id=29 and text_value like '%U.S. Agency for International Development%'; +UPDATE 3 +``` + +- I finally got a CGIAR account so I logged into CGSpace with it and tried to delete my old unfinished submissions (22 of them) +- Eventually it succeeded, but it took about five minutes and I noticed LOTS of locks happening with this query: + +``` +dspace=# \copy (SELECT * FROM pg_locks pl LEFT JOIN pg_stat_activity psa ON pl.pid = psa.pid) to /tmp/locks-aorth.txt; +``` + +- I took a few snapshots during the process and noticed 500, 800, and even 2000 locks at certain times during the process +- Afterwards I looked a few times and saw only 150 or 200 locks +- On the test server, with the [PostgreSQL indexes from DS-3636](https://jira.duraspace.org/browse/DS-3636) applied, it finished instantly diff --git a/docs/2018-02/index.html b/docs/2018-02/index.html index bc709b3d0..ae0c09f3c 100644 --- a/docs/2018-02/index.html +++ b/docs/2018-02/index.html @@ -23,7 +23,7 @@ I copied the logic in the jmx_tomcat_dbpools provided by Ubuntu’s munin-pl - + @@ -57,9 +57,9 @@ I copied the logic in the jmx_tomcat_dbpools provided by Ubuntu’s munin-pl "@type": "BlogPosting", "headline": "February, 2018", "url": "https://alanorth.github.io/cgspace-notes/2018-02/", - "wordCount": "5957", + "wordCount": "6400", "datePublished": "2018-02-01T16:28:54+02:00", - "dateModified": "2018-02-27T17:34:48+02:00", + "dateModified": "2018-02-27T18:50:30+02:00", "author": { "@type": "Person", "name": "Alan Orth" @@ -1193,6 +1193,113 @@ COPY 263
  • I will run it on DSpace Test
    • +

    2018-02-28

    + + + +
    # zcat --force /var/log/nginx/*.log /var/log/nginx/*.log.1 | grep -E "28/Feb/2018:09:" | awk '{print $1}' | sort | uniq -c | sort -n | tail -n 10
+     65 197.210.168.174
+     74 213.55.99.121
+     74 66.249.66.90
+     86 41.204.190.40
+    102 130.225.98.207
+    108 192.0.89.192
+    112 157.55.39.218
+    129 207.46.13.21
+    131 207.46.13.115
+    135 207.46.13.101
+
    + + + +
    2018-02-28 09:19:29,692 ERROR org.dspace.app.xmlui.cocoon.DSpaceCocoonServletFilter @ Serious Error Occurred Processing Request!
+org.springframework.web.util.NestedServletException: Handler processing failed; nested exception is java.lang.OutOfMemoryError: Java heap space
+
    + + + +
    $ grep -c 'nested exception is java.lang.OutOfMemoryError: Java heap space' dspace.log.2018-02-* 
+dspace.log.2018-02-01:0
+dspace.log.2018-02-02:0
+dspace.log.2018-02-03:0
+dspace.log.2018-02-04:0
+dspace.log.2018-02-05:0
+dspace.log.2018-02-06:0
+dspace.log.2018-02-07:0
+dspace.log.2018-02-08:0
+dspace.log.2018-02-09:0
+dspace.log.2018-02-10:0
+dspace.log.2018-02-11:0
+dspace.log.2018-02-12:0
+dspace.log.2018-02-13:0
+dspace.log.2018-02-14:0
+dspace.log.2018-02-15:10
+dspace.log.2018-02-16:0
+dspace.log.2018-02-17:0
+dspace.log.2018-02-18:0
+dspace.log.2018-02-19:0
+dspace.log.2018-02-20:0
+dspace.log.2018-02-21:0
+dspace.log.2018-02-22:0
+dspace.log.2018-02-23:0
+dspace.log.2018-02-24:0
+dspace.log.2018-02-25:0
+dspace.log.2018-02-26:0
+dspace.log.2018-02-27:6
+dspace.log.2018-02-28:1
+
    + + + +
    $ grep -E '2018-02-28 09:(0|1)' dspace.log.2018-02-28 | grep -o -E 'session_id=[A-Z0-9]{32}' | sort -n | uniq -c | sort -n | tail -n 10
+     18 session_id=F2DFF64D3D707CD66AE3A873CEC80C49
+     19 session_id=92E61C64A79F0812BE62A3882DA8F4BA
+     21 session_id=57417F5CB2F9E3871E609CEEBF4E001F
+     25 session_id=C3CD265AB7AA51A49606C57C069A902A
+     26 session_id=E395549F081BA3D7A80F174AE6528750
+     26 session_id=FEE38CF9760E787754E4480069F11CEC
+     33 session_id=C45C2359AE5CD115FABE997179E35257
+     38 session_id=1E9834E918A550C5CD480076BC1B73A4
+     40 session_id=8100883DAD00666A655AE8EC571C95AE
+     66 session_id=01D9932D6E85E90C2BA9FF5563A76D03
+
    + + + +
    cgspace=# update metadatavalue set text_value='United States Agency for International Development' where resource_type_id=2 and metadata_field_id=29 and text_value like '%U.S. Agency for International Development%';
+UPDATE 3
+
    + + + +
    dspace=# \copy (SELECT * FROM pg_locks pl LEFT JOIN pg_stat_activity psa ON pl.pid = psa.pid) to /tmp/locks-aorth.txt;
+
    + + + diff --git a/docs/robots.txt b/docs/robots.txt index 86c3f15b8..12f3bc2f9 100644 --- a/docs/robots.txt +++ b/docs/robots.txt @@ -32,7 +32,7 @@ Disallow: /cgspace-notes/2015-12/ Disallow: /cgspace-notes/2015-11/ Disallow: /cgspace-notes/ Disallow: /cgspace-notes/categories/ -Disallow: /cgspace-notes/categories/notes/ Disallow: /cgspace-notes/tags/notes/ +Disallow: /cgspace-notes/categories/notes/ Disallow: /cgspace-notes/post/ Disallow: /cgspace-notes/tags/ diff --git a/docs/sitemap.xml b/docs/sitemap.xml index 155758c73..5849b5834 100644 --- a/docs/sitemap.xml +++ b/docs/sitemap.xml @@ -4,7 +4,7 @@ https://alanorth.github.io/cgspace-notes/2018-02/ - 2018-02-27T17:34:48+02:00 + 2018-02-27T18:50:30+02:00 @@ -149,7 +149,7 @@ https://alanorth.github.io/cgspace-notes/ - 2018-02-27T17:34:48+02:00 + 2018-02-27T18:50:30+02:00 0 @@ -158,27 +158,27 @@ 0 + + https://alanorth.github.io/cgspace-notes/tags/notes/ + 2018-02-27T18:50:30+02:00 + 0 + + https://alanorth.github.io/cgspace-notes/categories/notes/ 2017-09-28T12:00:49+03:00 0 - - https://alanorth.github.io/cgspace-notes/tags/notes/ - 2018-02-27T17:34:48+02:00 - 0 - - https://alanorth.github.io/cgspace-notes/post/ - 2018-02-27T17:34:48+02:00 + 2018-02-27T18:50:30+02:00 0 https://alanorth.github.io/cgspace-notes/tags/ - 2018-02-27T17:34:48+02:00 + 2018-02-27T18:50:30+02:00 0