Faultinjector技術(shù)架構(gòu)介紹

1、Fa u l ti n j e c t i o nt ot e s tPo s t g r e S Q Lc o d eFaultinjector技術(shù)架構(gòu)介紹AgendaWhat is lacking in current testing frameworks?What is fault injection?Proposal to inject faultsTests to demonstrate proposed faultinjector patchTesting frameworks in PostgreSQLpg_regress (src/test/regress)A test is

2、written as a .sql fileExpected output is saved as an answer file (.out)pg_regress runs a .sql file, compares the output with the answerisolation (src/test/isolation)tests concurrent transactions.spec file - SQL commands within each transaction and how to interleave themisolation: test that select wa

3、its for altersession “s1”setup BEGIN ISOLATION LEVEL READ COMMITTED; step “alter1” ALTER TABLE test ADD RENAME TO test2; step “commit1” COMMIT; session “s2”setup SET default_transaction_isolation = read committed; step “select2” SELECT * from test; permutation “alter1” “s2” “commit1”Testing framewor

4、ks in PostgreSQLTAP - tests written in perl (Test:More)Orchestrate a cluster - master and one or more standbyinitdb, start / stop cluster, etc.src/test/perl/READMEsrc/test/modules (not run in CI)modular testing of a specific component, e.g. plannersrc/test/modules/READMEWhat is lacking in PostgreSQL

5、 tests?Test that a backend is killed after writing commit record but before updating the transaction status in pg_clogIs synchronous replication really synchronous?Was a cached plan reused?Anything comes to your mind?Fault injectionUnconference discussion at PGcon 2019 in OttawaAlready being used in

6、 GreenplumPatch proposed on pgsql-hackers:https:/www.postgr/message-id/flat/ CANXE4TdxdESX1jKw48xet-5GvBFVSq%3D4cgNeioTQff372KO45A%40Fault point (./configure CPPFLAGS=-DFAULT_INJECTOR)Fault point - macro definition-a/src/backend/access/heap/heapam.c+ b/src/backend/access/heap/heapam.cHeapTupletup,Co

7、mmandldcid, -1875,6 +1875,12Buffer bool heap_insert(Relation relation, vmbuffer = InvalidBuffer; all_visible_cleared = false;+#ifdefFAULT INJECTORFaultinjector_TriggerFaultifSet(11heap_insert11,+#endif1111/* databasename*/, RelationGetRelationName(relation);necessary.* Fill in tupleheader fields and

8、 toastthe tupleifSet a fault using its nameCREATE EXTENSION faultinjector;Enable the “heap_insert” fault when a tuple is inserted into “mytable”SELECT inject_fault(heap_insert, error, , mytable, );Wait until the fault is triggered (blocking call)SELECT wait_until_triggered_fault(heap_insert, 1);Chec

9、k the statusSELECT inject_fault(heap_insert, status);Reset the faultSELECT inject_fault(heap_insert, reset);More on inject_faultinject_fault( ,trigger only after the fault point is reached as many number of times,stop after triggerring as many number of times (fault state completed)seconds to sleep

10、in a sleep fault)inject_fault_remote()Same as inject_fault()Additional args: hostname and port numberNew libpq message to inject faultsUseful for testing standby servers in a clusterFault typeserror: elog(ERORR) leads to transaction abortskip: do nothing - indicates that the fault point was reached,

11、 also used for custom actionsuspendresetstatusflags)-a/src/backend/access/transam/xlog.c+ b/src/backend/access/transam/xlog.c -8537,6 +8537,14 Cr e a t ,eCh ec kPo i nt ( i nt Vi r t ua l T r a ns a c t i o n l d *vxids;intnvxids;+#ifdefFAULTINJECTOR+ if(SIMPLE_FAULT_INJECTOR(11checkpoint11)Faultinj

12、ector_FaultTypeSkip)I*Custom logichere.*Ireturn;+#endifisreallya shutdown checkpoint,justAn end-of-recoverycheckpointissuedata differenttime.Fault statesinjected but not triggered - SELECT inject_fault();completed - triggered maximum number of times, will no longer triggertriggered - reached during

13、execution at least onceFaults offer fine grained controlEnable complex testing scenariosWas a specific flag in shared memory set, and when?Was a branch in a function taken?Can be used in regress, isolation as well as TAP testsDemonstrative test: speculative insertCREATE TABLE test(key TEXT, data TEX

14、T);CREATE UNIQUE INDEX ON test(key);T1: INSERT INTO test values (k1, inserted T1) ON CONFLICT DO UPDATE SET data = test.data | conflict update T1;T2: INSERT INTO test values (k1, inserted T2) ON CONFLICT DO UPDATE SET data = test.data | conflict update T2;Demonstrative test: speculative insertConfli

15、cts detected after a tuple is inserted into heap but before inserting into index are handled properlyT1 goes first and inserts the tuple into heapBefore T1 updates the index, T2 inserts into both, heap and the index.T2 sees no conflicts because T1 is still in progress.T1 moves ahead.T1 should detect

16、 a conflict and abort.Demonstrative test: speculative insertFind it in the faultinjector patch:src/test/isolation/specs/insert-conflict-with-faults.specDemonstrative test: synchronous replicationEnsure that commits on master block until a synchronous standby acknowledges the commitMaster writes a commit recordThe backend process on master starts wai