↑日記で日々積み重ねた情報をトップの「わんこのページ」にまとめています。

わんこ日記 [RDF] YouTubeも始めました→


2019-10-25(Fri) [GitLab] GitLab 12.4のupgradeで失敗 [長年日記] この日を編集

GitLabは、OmnibusパッケージのDocker imageにしてから、upgradeに失敗することは無かったのだけど、12.4に上げたら

itlab-ce_1 | * bash[migrate gitlab-rails database] action run

gitlab-ce_1 | [execute] rake aborted!

gitlab-ce_1 | StandardError: An error has occurred, all later migrations canceled:

gitlab-ce_1 |

gitlab-ce_1 | Multiple indexes found on ci_trigger_requests columns [:trigger_id]. Specify an index name from ci_trigger_requests_trigger_id_idx, fk_rails_b8ec8b7245, index_ci_trigger_requests_on_trigger_id

gitlab-ce_1 | /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers.rb:121:in `block in remove_concurrent_index'

で再起動を繰り返していた。

どうも「12.2.5 -> 12.3.1 Database migration failed: Multiple indexes found (#33084) · Issues · GitLab.org / GitLabの現象に近そうなので、インデックスを削除したらうまく行った。

$ docker-compose exec gitlab-ce bash

oot@gitlab-ce:/# gitlab-psql

psql (10.9)

Type "help" for help.

gitlabhq_production=# /d ci_trigger_requests

Table "public.ci_trigger_requests"

Column | Type | Collation | Nullable | Default

------------+--------------------------+-----------+----------+-------------------------------------------------

id | integer | | not null | nextval('ci_trigger_requests_id_seq'::regclass)

trigger_id | integer | | not null |

variables | text | | |

created_at | timestamp with time zone | | |

updated_at | timestamp with time zone | | |

commit_id | integer | | |

Indexes:

"ci_trigger_requests_pkey" PRIMARY KEY, btree (id)

"ci_trigger_requests_trigger_id_idx" btree (trigger_id)

"fk_rails_b8ec8b7245" btree (trigger_id)

"index_ci_trigger_requests_on_commit_id" btree (commit_id)

"index_ci_trigger_requests_on_trigger_id" btree (trigger_id)

"index_ci_trigger_requests_on_trigger_id_and_id" btree (trigger_id, id DESC)

Foreign-key constraints:

"fk_rails_b8ec8b7245" FOREIGN KEY (trigger_id) REFERENCES ci_triggers(id) ON DELETE CASCADE DEFERRABLE INITIALLY DEFERRED

gitlabhq_production=# drop index ci_trigger_requests_trigger_id_idx;

DROP INDEX

gitlabhq_production=# drop index index_ci_trigger_requests_on_trigger_id;

DROP INDEX

MySQLから無理くりPostgreSQLにデータをコンバートした弊害か?


2019-02-26(Tue) Tiny Tyin RSS画像大きすぎ [長年日記] この日を編集

TT-RSSのプラグインには1280pxという添付の画像を大きくするものはあるけど、JBpressとか東洋経済オンラインのRSSに付いてくる画像は大きすぎ、defaultではmax-width: 98%だけど、設定のカスタムRSSで

div.content > div.content-inner img {

max-width: 50%;

}

div.content > div.intermediate img {

max-width: 50%;

}

を追加して、画像を小さくした。


2019-02-11(Mon) tt-rss 18へupgradeで画面でなくなる [長年日記] この日を編集

Debian Packetのupgradeに伴い tt-rssを17.4から18.12にupgradeしたら、phpmailerは使わなくなり、Debian Pakcege依存での問題はなくなったのだけど、Webアクセスに行くと500エラーとなる。

[Tue Feb 11 13:26:08.708664 2019] [proxy_fcgi:error] [pid 13904:tid 139789892630272] [client 127.0.0.1:197] AH01071: Got error 'PHP message: PHP Fatal error: Uncaught PDOException: SQLSTATE[22001]: String data, right truncated: 1406 Data too long for column 'context' at row 1 in /usr/share/tt-rss/www/classes/logger/sql.php:18

Stack trace:

#0 /usr/share/tt-rss/www/classes/logger/sql.php(18): PDOStatement->execute(Array)

#1 /usr/share/tt-rss/www/classes/logger.php(28): Logger_SQL->log_error(8192, 'strpos(): Non-s...', 'vendor/JShrink/...', 184, '1. vendor/JShri...')

#2 /usr/share/tt-rss/www/include/errorhandler.php(43): Logger->log_error(8192, 'strpos(): Non-s...', 'vendor/JShrink/...', 184, '1. vendor/JShri...')

#3 [internal function]: ttrss_error_handler(8192, 'strpos(): Non-s...', 'vendor/JShrink/...', 184, '1. vendor/JShri...')\n

#4 /usr/share/tt-rss/www/vendor/JShrink/Minifier.php(184): strpos('(-+{[@', false)

#5 /usr/share/tt-rss/www/vendor/JShrink/Minifier.php(144): JShrink\\Minifier->loop()

#6 /usr/share/tt-rss/www/vendor/JShrink/Minifier.php(110): JShrink\\Minifier->minifyDirectToOutput('/* global dijit...', Array)

#7 ...'

というエラー。

「1406 Data too long for column 'context'」ということでどうやらLogを記録するために error_log に書き込む時にログメッセージであるcontextが長すぎたようなのでsubstr()で、MySQL text型の最長の65536文字に制限するバッチを/usr/share/tt-rss/www/classes/logger/sql.phpに当てた

--- sql.php.org 2018-11-30 23:02:05.000000000 +0900

+++ sql.php 2019-02-11 14:29:30.216183169 +0900

@@ -15,7 +15,7 @@

$sth = $this->pdo->prepare("INSERT INTO ttrss_error_log

(errno, errstr, filename, lineno, context, owner_uid, created_at) VALUES

(?, ?, ?, ?, ?, ?, NOW())");

- $sth->execute([$errno, $errstr, $file, $line, $context, $owner_uid]);

+ $sth->execute([$errno, $errstr, $file, $line, substr($context,0,65535), $owner_uid]);

return $sth->rowCount();

}


2019-02-05(Tue) [長年日記] この日を編集

Apache2でHTTP/2が止まっていた

一度有効にしたはずなApache2のHTTP/2なのだけど、

[Mon Feb 04 23:18:04.440099 2019] [http2:warn] [pid 25940] AH10034: The mpm module (prefork.c) is not supported by mod_http2. The mpm determines how things are processed in your server. HTTP/2 has more demands in this regard and the currently selected mpm will just not do. This is an advisory warning. Your server will continue to work, but the HTTP/2 protocol will be inactive.

という警告が出て、mpm_preforkでは使えなくなっていた。

mpmの切替は以前(って10年近く前?)に挑戦して余り印象が良くないのだけど、重要なサービスはDockerコンテナに移行していて、PHP CGIが変わる以外は影響が少なさそうなのでmpm_eventに移行してHTTP/2を有効にする。

mod_php7.3は、mpm_preforkが前提なのでPHP CGIをfcgidとphp7.3-fpmに変える。

# apt-get install php-fpm

# a2dismod php7.3 mpm_prefork

# a2enmod mpm_event fcgid proxy_fcgi

# a2enconf php7.3-fpm

途中、proxy_fcgiをenableにしていず、

[Mon Feb 04 23:20:03.788097 2019] [proxy:warn] [pid 10814] [client 54.36.149.80:28334] AH01144: No protocol handler was valid for the URL /wiki/index.php (scheme 'unix'). If you are using a DSO version of mod_proxy, make sure the proxy submodules are included in the configuration using LoadModule.

というエラーが出て嵌ったけれど、php7.3-fpm.confでproxy_fcgiが無いと有効にならないことに気がついて見つけた。

Apache2でHTTP/2が止まっていた

一度有効にしたはずなApache2のHTTP/2なのだけど、

[Mon Feb 04 23:18:04.440099 2019] [http2:warn] [pid 25940] AH10034: The mpm module (prefork.c) is not supported by mod_http2. The mpm determines how things are processed in your server. HTTP/2 has more demands in this regard and the currently selected mpm will just not do. This is an advisory warning. Your server will continue to work, but the HTTP/2 protocol will be inactive.

という警告が出て、mpm_preforkでは使えなくなっていた。

mpmの切替は以前(って10年近く前?)に挑戦して余り印象が良くないのだけど、重要なサービスはDockerコンテナに移行していて、PHP CGIが変わる以外は影響が少なさそうなのでmpm_eventに移行してHTTP/2を有効にする。

mod_php7.3は、mpm_preforkが前提なのでPHP CGIをfcgidとphp7.3-fpmに変える。

# apt-get install php-fpm

# a2dismod php7.3 mpm_prefork

# a2enmod mpm_event fcgid proxy_fcgi

# a2enconf php7.3-fpm

途中、proxy_fcgiをenableにしていず、

[Mon Feb 04 23:20:03.788097 2019] [proxy:warn] [pid 10814] [client 54.36.149.80:28334] AH01144: No protocol handler was valid for the URL /wiki/index.php (scheme 'unix'). If you are using a DSO version of mod_proxy, make sure the proxy submodules are included in the configuration using LoadModule.

というエラーが出て嵌ったけれど、php7.3-fpm.confでproxy_fcgiが無いと有効にならないことに気がついて見つけた。



2018-11-15(Thu) [長年日記] この日を編集

[ZFS][HDD] 結果

昨日のZFSミラーのリビルドは無事に終わって、最大値は1023IOPS Throughput 123.14Mbype/secだった。

画像の説明画像の説明


2018-11-14(Wed) [長年日記] この日を編集

[ZFS][HDD] ディスク交換 3TB→8TB

先日の日記のようにHDDが壊れたようで8セクターが読めなくなったのでHDD交換をする。

従来は、3TBと4TBのzfs mirrorで、生き残った4TBを最大容量使っていないので、ミラーの生きている方を最大容量使え切るようにpartedで1MiB単位でzfs poolを最大化してからHDDを物理的に入れ替えた。

マルチユーザでブートしてから、gpartedでパテーションテーブル作ったりパティーションを未フォーマットで切った。

このへんは、昔と違ってサーバをマルチユーザで完全に起動した状態になってからGUIベースでできるので助かる。

で、zfsの玉のリプレース。リプレースするとミラーのリビルドも自動的に走る。

# zpool replace tank 2189429116797733421 ata-TOSHIBA_MD05ACA800_ZZRCK0MSZZZZ-part1

# zpool status -v

pool: tank

state: DEGRADED

status: One or more devices is currently being resilvered. The pool will

continue to function, possibly in a degraded state.

action: Wait for the resilver to complete.

scan: resilver in progress since Tue Nov 13 23:50:17 2018

19.2G scanned out of 1.79T at 16.1M/s, 32h10m to go

19.2G resilvered, 1.04% done

config:

NAME STATE READ WRITE CKSUM

tank DEGRADED 0 0 0

mirror-0 DEGRADED 0 0 0

ata-ST4000DM000-1F2168_Z307V2E2-part1 ONLINE 0 0 0

replacing-1 DEGRADED 0 0 0

2189429116797733421 UNAVAIL 0 0 0 was /dev/disk/by-id/ata-TOSHIBA_DT01ACA300_66SWYX9FS-part1

ata-TOSHIBA_MD05ACA800_ZZRCK0MSZZZZ-part1 ONLINE 0 0 0 (resilvering)

errors: No known data errors

ってわけで、ミラーのリビルドには一日かかる模様。


2018-11-12(Mon) [長年日記] この日を編集

[ZFS][HDD] HDDやはり壊れた

ZFSからエラーのメールを頂いていたのを、昨日の日記で書いたけど、今日はSMARTからもメールが来た。

# smartctl -l error /dev/sdb

smartctl 6.6 2017-11-05 r4594 [x86_64-linux-4.18.0-2-amd64] (local build)

Copyright (C) 2002-17, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF READ SMART DATA SECTION ===

SMART Error Log Version: 1

ATA Error Count: 2

CR = Command Register [HEX]

FR = Features Register [HEX]

SC = Sector Count Register [HEX]

SN = Sector Number Register [HEX]

CL = Cylinder Low Register [HEX]

CH = Cylinder High Register [HEX]

DH = Device/Head Register [HEX]

DC = Device Command Register [HEX]

ER = Error register [HEX]

ST = Status register [HEX]

Powered_Up_Time is measured from power on, and printed as

DDd+hh:mm:SS.sss where DD=days, hh=hours, mm=minutes,

SS=sec, and sss=millisec. It "wraps" after 49.710 days.

Error 2 occurred at disk power-on lifetime: 42730 hours (1780 days + 10 hours)

When the command that caused the error occurred, the device was active or idle.

After command completion occurred, registers were:

ER ST SC SN CL CH DH

-- -- -- -- -- -- --

40 51 d8 a0 a2 d4 0f Error: UNC 216 sectors at LBA = 0x0fd4a2a0 = 265593504

Commands leading to the command that caused the error were:

CR FR SC SN CL CH DH DC Powered_Up_Time Command/Feature_Name

-- -- -- -- -- -- -- -- ---------------- --------------------

25 00 00 78 a2 d4 e0 08 3d+14:16:29.677 READ DMA EXT

35 00 58 50 e7 79 e0 08 3d+14:16:29.677 WRITE DMA EXT

b0 d0 01 00 4f c2 00 08 3d+14:16:29.489 SMART READ DATA

35 00 90 e0 75 48 e0 08 3d+14:16:29.489 WRITE DMA EXT

ef 10 02 00 00 00 a0 08 3d+14:16:29.489 SET FEATURES [Enable SATA feature]

Error 1 occurred at disk power-on lifetime: 42722 hours (1780 days + 2 hours)

When the command that caused the error occurred, the device was active or idle.

After command completion occurred, registers were:

ER ST SC SN CL CH DH

-- -- -- -- -- -- --

84 51 ba 46 83 61 08 Error: ICRC, ABRT at LBA = 0x08618346 = 140608326

Commands leading to the command that caused the error were:

CR FR SC SN CL CH DH DC Powered_Up_Time Command/Feature_Name

-- -- -- -- -- -- -- -- ---------------- --------------------

61 00 80 00 83 61 40 08 3d+06:13:43.240 WRITE FPDMA QUEUED

61 00 50 00 82 61 40 08 3d+06:13:43.240 WRITE FPDMA QUEUED

61 00 78 00 81 61 40 08 3d+06:13:43.239 WRITE FPDMA QUEUED

61 a0 d8 10 d8 35 40 08 3d+06:13:43.239 WRITE FPDMA QUEUED

61 80 48 50 73 10 40 08 3d+06:13:43.239 WRITE FPDMA QUEUED

というとこなので、2013年12月の日記から、5年近く使ったHDDも交換だね。3TB→8TBにする。


2018-11-11(Sun) [ZFS] DISKが壊れた? [長年日記] この日を編集

ZFSでエラー発生メールを受け取る

From: root

Subject: ZFS device fault for pool 0x9773EB28D655981A on on-o.com

The number of I/O errors associated with a ZFS device exceeded

acceptable levels. ZFS has marked the device as faulted.

impact: Fault tolerance of the pool may be compromised.

eid: 40

class: statechange

state: FAULTED

host: on-o.com

time: 2018-11-11 07:25:07+0900

vpath: /dev/disk/by-id/ata-TOSHIBA_DT01ACA300_66WYX9FS-part1

vguid: 0x1E626A2E7FA04A2D

pool: 0x9773EB28D655981A

マジですか? と思っていたら自動的にscrubも実行されて、メールが来た

Subject: ZFS scrub_finish event for tank on on-o.com

ZFS has finished a scrub:

eid: 43

class: scrub_finish

host: on-o.com

time: 2018-11-11 11:00:20+0900

pool: tank

state: DEGRADED

status: One or more devices are faulted in response to persistent errors.

Sufficient replicas exist for the pool to continue functioning in a

degraded state.

action: Replace the faulted device, or use 'zpool clear' to mark the device

repaired.

scan: scrub repaired 288K in 10h36m with 0 errors on Sun Nov 11 11:00:20 2018

config:

NAME STATE READ WRITE CKSUM

tank DEGRADED 0 0 0

mirror-0 DEGRADED 0 0 0

ata-ST4000DM000-1F2169_Z997V2F2-part1 ONLINE 0 0 0

ata-TOSHIBA_DT01ACA300_66SWYX9FS-part1 FAULTED 0 0 9 too many errors

errors: No known data errors

というわけで、TOSHIBAの3TB HDDでチェックサムエラーが9回発生して、too many errorsでFAULTEDになっている。

詳しく見てみると

# zpool status -v

pool: tank

state: DEGRADED

status: One or more devices are faulted in response to persistent errors.

Sufficient replicas exist for the pool to continue functioning in a

degraded state.

action: Replace the faulted device, or use 'zpool clear' to mark the device

repaired.

scan: scrub repaired 288K in 10h36m with 0 errors on Sun Nov 11 11:00:20 2018

config:

NAME STATE READ WRITE CKSUM

tank DEGRADED 0 0 0

mirror-0 DEGRADED 0 0 0

ata-ST4000DM000-1F2169_Z997V2F2-part1 ONLINE 0 0 0

ata-TOSHIBA_DT01ACA300_66SWYX9FS-part1 FAULTED 0 0 9 too many errors

errors: No known data errors

「zpool clear」してみろとのことなので、まずはSMARTと並行してエラーのクリアをしてみる

# zpool clear tank ata-TOSHIBA_DT01ACA300_66SWYX9FS-part1

# zpool status

pool: tank

state: ONLINE

status: One or more devices is currently being resilvered. The pool will

continue to function, possibly in a degraded state.

action: Wait for the resilver to complete.

scan: resilver in progress since Mon Nov 12 22:53:58 2018

338G scanned out of 1.79T at 858M/s, 0h29m to go

5.44G resilvered, 18.37% done

config:

NAME STATE READ WRITE CKSUM

tank ONLINE 0 0 0

mirror-0 ONLINE 0 0 0

ata-ST4000DM000-1F2169_Z997V2F2-part1 ONLINE 0 0 0

ata-TOSHIBA_DT01ACA300_66SWYX9FS-part1 ONLINE 0 0 0 (resilvering)

errors: No known data errors

29分待てば良いのかと思いきや、いま見ると3時間待ちなので、SMARTテストも同時実行する。

SMART shortテスト

# smartctl -t short /dev/sdb

smartctl 6.6 2017-11-05 r4594 [x86_64-linux-4.18.0-2-amd64] (local build)

Copyright (C) 2002-17, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF OFFLINE IMMEDIATE AND SELF-TEST SECTION ===

Sending command: "Execute SMART Short self-test routine immediately in off-line mode".

Drive command "Execute SMART Short self-test routine immediately in off-line mode" successful.

Testing has begun.

Please wait 1 minutes for test to complete.

Test will complete after Mon Nov 12 22:47:49 2018

Use smartctl -X to abort test.

# smartctl -l error /dev/sdb

smartctl 6.6 2017-11-05 r4594 [x86_64-linux-4.18.0-2-amd64] (local build)

Copyright (C) 2002-17, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF READ SMART DATA SECTION ===

SMART Error Log Version: 1

No Errors Logged

# smartctl -l selftest /dev/sdb

smartctl 6.6 2017-11-05 r4594 [x86_64-linux-4.18.0-2-amd64] (local build)

Copyright (C) 2002-17, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF READ SMART DATA SECTION ===

SMART Self-test log structure revision number 1

Num Test_Description Status Remaining LifeTime(hours) LBA_of_first_error

# 1 Short offline Completed without error 00% 42719 -

# 2 Short offline Completed without error 00% 32323 -

# 3 Extended offline Aborted by host 90% 32323 -

# 4 Short offline Completed without error 00% 25958 -

# 5 Short offline Completed without error 00% 25958 -

とりあえず、shortテストはOK

SMART offlineテスト

# smartctl -t offline /dev/sdb

smartctl 6.6 2017-11-05 r4594 [x86_64-linux-4.18.0-2-amd64] (local build)

Copyright (C) 2002-17, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF OFFLINE IMMEDIATE AND SELF-TEST SECTION ===

Sending command: "Execute SMART off-line routine immediately in off-line mode".

Drive command "Execute SMART off-line routine immediately in off-line mode" successful.

Testing has begun.

Please wait 23082 seconds for test to complete.

Test will complete after Tue Nov 13 05:12:42 2018

Use smartctl -X to abort test.

朝までかかるので、果報は寝て待つ


2018-06-06(Wed) [長年日記] この日を編集

Debian bugreportしました

昨日の日記のStunnel4で、compressionが使えなくなった件をバグレポートした。

Debian Bug report logs - #900896

stunnel4: "compression" configuration occurred an error

本日のツッコミ(全1件) [ツッコミを入れる]

_ わんこ [何のことない、OpenSSLで圧縮サポートされなくなってた]


Google Web検索 on-o.com内を検索