Thursday, August 26, 2010

Column Encryption using DBMS_CRYPTO


Oracle Database 10g provides means to encrypt and decrypt your sensitive data using the built-in API called DBMS_CRYPTO. Using DBMS_CRYPTO API one can encrypt sensitive information like SSN, Credit Card Numbers, Debit Card Numbers, etc stored in the database as these are not supposed to be stored in plain text for security reasons.

Let’s  see how DBMS_CRYPTO can be used to encrypt data and also consider its performance implications.

create table emp(empid number, ename varchar2(100), ssn varchar2(80)); 
Table created.    

insert into emp values (1, 'SMITH',        123456701);    
1 row created. 
insert into emp values (2, 'ALLEN',        123456702);
 1 row created. 
 insert into emp values (3, 'WARD',         123456703); 
 1 row created. 
commit; 

select * from emp; 
      EMPID ENAME                          SSN 
 ---------- ------------------------------ ------------- 
          1 SMITH                          123456701 
          2 ALLEN                          123456702 
          3 WARD                           123456703

The SSN has been inserted into the table in plain text and can be read by anybody having SELECT privilege on EMP table.

Now, let’s create a package using DBMS_CRYPTO API to handle encryption and decryption.

Create or replace package cryptit is 
Function encrypt_data( p_data IN VARCHAR2 ) Return RAW DETERMINISTIC; 
Function decrypt_data( p_data IN RAW ) Return VARCHAR2 DETERMINISTIC; 
End cryptit; 
  
 Package created. 
  
Create or replace package body cryptit is 
V_Key       RAW(128) := UTL_RAW.cast_to_raw('testkey1');          -- Key 
  Function encrypt_data( p_data IN VARCHAR2 ) Return RAW DETERMINISTIC 
    IS 
   l_data RAW(2048) := utl_raw.cast_to_raw(p_data); 
   l_encrypted RAW(2048); 
   BEGIN 
   NULL; 
   l_encrypted := dbms_crypto.encrypt                        -- Algorithm 
( src => l_data, 
                 typ => DBMS_CRYPTO.DES_CBC_PKCS5, 
   key => V_KEY ); 
   Return l_encrypted; 
END encrypt_data; 
Function decrypt_data( p_data IN RAW ) Return VARCHAR2 DETERMINISTIC 
   IS 
l_decrypted RAW(2048); 
BEGIN 
l_decrypted := dbms_crypto.decrypt                              -- Algorithm 
 ( src => p_data, 
typ => DBMS_CRYPTO.DES_CBC_PKCS5, 
key => V_KEY ); 
Return utl_raw.cast_to_varchar2(l_decrypted); 
END decrypt_data; 
End cryptit; 

Package body created. 

The package body consists of functions to encrypt and decrypt data along with a key. As these functions will always output the same value for a specific input, we can make them Deterministic functions. (A Deterministic Function always returns the same result any time they are called with a specific set of input values.)

Let's now make use of these functions to encrypt our sensitive data.

update emp set ssn = cryptit.encrypt_data(ssn); 
3 rows updated. 

select * from emp; 
      EMPID ENAME                          SSN 
---------- ------------------------------ ---------------------------------- 
         1 SMITH                          5F3168C22E54060DE7D97B31F7E38BB6 
         2 ALLEN                          5F3168C22E54060D0C2527FBBD3DCD6C 
         3 WARD                           5F3168C22E54060D8166D3757932A112 

When inserting new records we simply make use of "cryptit.encrypt_data" function in the INSERT statement.

insert into emp values( 4, 'NADVI', cryptit.encrypt_data(123456704)); 
1 row created. 
 commit; 
#  
Commit complete. 

select * from emp; 
      EMPID ENAME                          SSN 
---------- ------------------------------ ---------------------------------------- 
          1 SMITH                          5F3168C22E54060DE7D97B31F7E38BB6 
          2 ALLEN                          5F3168C22E54060D0C2527FBBD3DCD6C 
          3 WARD                           5F3168C22E54060D8166D3757932A112 
          4 NADVI                          5F3168C22E54060D2CFF7E7A35B14187 
  

 select empid, ename, cryptit.decrypt_data(ssn) decrypted_ssn, ssn from emp; 

     EMPID ENAME                          DECRYPTED_SSN   SSN 
 ---------- ------------------------------ --------------- ---------------------------------------- 
          1 SMITH                          123456701       5F3168C22E54060DE7D97B31F7E38BB6 
          2 ALLEN                          123456702       5F3168C22E54060D0C2527FBBD3DCD6C 
          3 WARD                           123456703       5F3168C22E54060D8166D3757932A112 
          4 NADVI                          123456704       5F3168C22E54060D2CFF7E7A35B14187 

All is fine as far as encryption is concerned, but lets see what happens when SSN is used in the WHERE clause of queries.

We will create an index on SSN and run a query against it.

create index t_ssn on emp(ssn);
Index created.

 select * from emp where ssn = cryptit.encrypt_data('123456701');

     EMPID ENAME                          SSN
---------- ------------------------------ ----------------------------------------
         1 SMITH                          5F3168C22E54060DE7D97B31F7E38BB6

Execution Plan
----------------------------------------------------------
Plan hash value: 2894032564

-------------------------------------------------------------------------------------
| Id  | Operation                   | Name  | Rows  | Bytes | Cost (%CPU)| Time     |
-------------------------------------------------------------------------------------
|   0 | SELECT STATEMENT            |       |     1 |   107 |     2   (0)| 00:00:01 |
|   1 |  TABLE ACCESS BY INDEX ROWID| EMP   |     1 |   107 |     2   (0)| 00:00:01 |
|*  2 |   INDEX RANGE SCAN          | T_SSN |     1 |       |     1   (0)| 00:00:01 |
-------------------------------------------------------------------------------------

Predicate Information (identified by operation id):
---------------------------------------------------

   2 - access("SSN"=RAWTOHEX("CRYPTIT"."ENCRYPT_DATA"('123456701')))
Note
-----
   - dynamic sampling used for this statement

Well, the index "T_SSN" has been selected by the optimizer to be cost efficient. When a range of values is to be scanned, optimizer falls flat on its face as it has no idea and picks up FTS.

select * from emp where cryptit.decrypt_data(ssn) between '123456702' and '123456704';

     EMPID ENAME                          SSN
---------- ------------------------------ ----------------------------------------
         2 ALLEN                          5F3168C22E54060D0C2527FBBD3DCD6C
         3 WARD                           5F3168C22E54060D8166D3757932A112
         4 NADVI                          5F3168C22E54060D2CFF7E7A35B14187

Execution Plan
----------------------------------------------------------
Plan hash value: 3956160932
--------------------------------------------------------------------------
| Id  | Operation         | Name | Rows  | Bytes | Cost (%CPU)| Time     |
--------------------------------------------------------------------------
|   0 | SELECT STATEMENT  |      |     1 |   107 |     3   (0)| 00:00:01 |
|*  1 |  TABLE ACCESS FULL| EMP  |     1 |   107 |     3   (0)| 00:00:01 |
--------------------------------------------------------------------------
Predicate Information (identified by operation id):
---------------------------------------------------
   1 - filter("CRYPTIT"."DECRYPT_DATA"(HEXTORAW("SSN"))>='123456702'
              AND "CRYPTIT"."DECRYPT_DATA"(HEXTORAW("SSN"))<='123456704')
Note
-----
   - dynamic sampling used for this statement

We can also create Function Based Indexes on encrypted columns such that data is accessed faster with a better execution plan. This is the reason I have marked both ENCRYPT_DATA and DECRYPT_DATA as DETERMINISTIC functions.

create index f_ssn_idx on emp(cryptit.decrypt_data(ssn));

Index created.

select * from emp where cryptit.decrypt_data(ssn) between '123456702' and '123456704';

     EMPID ENAME                          SSN
---------- ------------------------------ ----------------------------------------
         2 ALLEN                          5F3168C22E54060D0C2527FBBD3DCD6C
         3 WARD                           5F3168C22E54060D8166D3757932A112
         4 NADVI                         5F3168C22E54060D2CFF7E7A35B14187


Execution Plan
----------------------------------------------------------
Plan hash value: 9274740

-----------------------------------------------------------------------------------------
| Id  | Operation                   | Name      | Rows  | Bytes | Cost (%CPU)| Time     |
-----------------------------------------------------------------------------------------
|   0 | SELECT STATEMENT            |           |     1 |   107 |     2   (0)| 00:00:01 |
|   1 |  TABLE ACCESS BY INDEX ROWID| EMP       |     1 |   107 |     2   (0)| 00:00:01 |
|*  2 |   INDEX RANGE SCAN          | F_SSN_IDX |     1 |       |     1   (0)| 00:00:01 |
-----------------------------------------------------------------------------------------

Predicate Information (identified by operation id):
---------------------------------------------------

   2 - access("CRYPTIT"."DECRYPT_DATA"(HEXTORAW("SSN"))>='123456702' AND
              "CRYPTIT"."DECRYPT_DATA"(HEXTORAW("SSN"))<='123456704')
Note
-----
   - dynamic sampling used for this statement

The optimizer has rightly picked up the Function based index to access requested data quickly.

DBMS_CRYPTO is a great way to encrypt sensitive data and we can also create indexes on the encrypted columns to speedup our queries. Lastly, remember to wrap the CRYPTIT package body so that the key is not exposed.

DBMS_CRYPTO can also be used to encrypted data recursively. Let us try to encrypt the already encrypted SSN from the above example.

select * from emp;

     EMPID ENAME    SSN
---------- -------- --------------------------------------------------------------------------------
         1 SMITH    5F3168C22E54060DE7D97B31F7E38BB6
         2 ALLEN    5F3168C22E54060D0C2527FBBD3DCD6C
         3 WARD     5F3168C22E54060D8166D3757932A112
         4 NADVI    5F3168C22E54060D2CFF7E7A35B14187

update emp set ssn = cryptit.encrypt_data(ssn);

4 rows updated.

select * from emp;

     EMPID ENAME    SSN
---------- -------- --------------------------------------------------------------------------------
         1 SMITH    455E357D80B1C81F91D52088DA0FE03A0C133D129CFC3879EE7A8DE993FD459DB5C9E1EBEF37062C
         2 ALLEN    455E357D80B1C81F91D52088DA0FE03A98FC410A71CE64D57935691692D433B770C5F8AF07C9113F
         3 WARD     455E357D80B1C81F91D52088DA0FE03A9D32D14FE64F41DD66BE7FEF4C31384F032C16826EAA4830
         4 NADVI    455E357D80B1C81F91D52088DA0FE03A04D5C0AD79E689EABD46F8EC335308E32D6F7E94167F6EDF

commit;

Commit complete.

select empid, ename, cryptit.decrypt_data(cryptit.decrypt_data(ssn)) decrypted_ssn, ssn from emp;

     EMPID ENAME    DECRYPTED_SSN   SSN
---------- -------- --------------- --------------------------------------------------------------------------------
         1 SMITH    123456701       455E357D80B1C81F91D52088DA0FE03A0C133D129CFC3879EE7A8DE993FD459DB5C9E1EBEF37062C
         2 ALLEN    123456702       455E357D80B1C81F91D52088DA0FE03A98FC410A71CE64D57935691692D433B770C5F8AF07C9113F
         3 WARD     123456703       455E357D80B1C81F91D52088DA0FE03A9D32D14FE64F41DD66BE7FEF4C31384F032C16826EAA4830
         4 NADVI    123456704       455E357D80B1C81F91D52088DA0FE03A04D5C0AD79E689EABD46F8EC335308E32D6F7E94167F6EDF

Friday, August 20, 2010

Tablespace NOLOGGING and objects

What happens with a NOLOGGING tablespace ? Will all the tables ( segments) created in that tablespace will not be logged by default?

By default the Tables (created in this Tablespace) will inherit the option NOLOGGING. It can be overridden at the Table level if you specify the option LOGGING when you create it.
So you can have Tables with the option LOGGING in a Tablespace with the option NOLOGGING.


The only reason to create a tablespace with nologging, or to later set the nologging attribute on a tablespace, is to change the default of all objects (segments) created in that tablespace from that point forward. It has no effect at all on segments that already exist in the tablespace.
That's it. It has no effect on how long it will take the tablespace to be created, or recoverability of that tablespace.

At the table/index level, it will only have an effect for direct load operations, such as insert /*+ append */, CTAS, or alter table move on a table, or index creation or rebuild.

Besides this, you have also the FORCE LOGGING /NO FORCE LOGGING mode.
The FORCE LOGGING / NO FORCE LOGGING is used to control at the Database or Tablespace level the Logging mode. By default the Database and Tablespaces are in NO FORCE LOGGING.
 The FORCE LOGGING mode always overrides the NOLOGGING option at the Tablespace or Table level.

So if you have the Database in FORCE LOGGING mode, your Tables will generates Redo logs even if these Tables are created with the option NOLOGGING.

It can be useful in a Data Guard so as to enforce Redo log generation to maintain the Standby Databases.

ORA-32006: %s initialization parameter has been deprecated


ORA-32006: %s initialization parameter has been deprecated
ORA-19905: log_archive_format must contain %s, %t and %r


while startup(Archivelog mode) the following message dispalyed

ORA-32006: LOG_ARCHIVE_START initialization parameter has been deprecated.

how to solve this problem ?
Just remove this parameter from your Initialization Parameter File (INIT.ORA/ SPFILE).

log_archive_start parameter is deprecated from 10g. LOG_ARCHIVE_START parameter is no longer required to be set in order to run the database in ARCHIVELOG mode. This parameter is deprecated in Oracle 10g. If you put this parameter in your init of spfile, database will start, but will get following error.

ORA-32006: LOG_ARCHIVE_START initialization parameter has been deprecated ORACLE instance started.

However, if we change the LOG_ARCHIVE_FORMAT to something other than the default %s,%t,%r, it will fail. and you get following error.

ORA-32006: LOG_ARCHIVE_START initialization parameter has been deprecated
ORA-19905: log_archive_format must contain %s, %t and %r

Could not startup the database in mount / nomount stage using Spfile as Spfile had wrong format for archive log.

To implement the solution, please execute the following steps:

1. Create a copy of the init.ora file. ( oracle_home/admin/pfile --> init.ora )
2. Edit the above init.ora to reflect the log_archive_format = 'arch_%r_%t_%s.arc'
and log_archive_dest_1 = f:\archive
3. Shutdown immediate and Startup nomount with the above pfile.
shutdown immediate
startup nomount pfile = oracle_home/admin/pfile/init.ora

4. Create spfile from pfile
create spfile = 'oracle_home/database/spfile.ora' from pfile ='oracle_home/admin/pfile/init.ora'

5. shutdown immediate
6. startup
7. archive log list --- verify that db is in arhivelog mode. 

Enabling Archive Log mode for Single Node Instance & RAC


On a single node instance the archive log mode is reset as follows:

SQL> archive log list
Database log mode                         No Archive Mode
Automatic archival                           Disabled
Archive destination                         USE_DB_RECOVERY_FILE_DEST
Oldest online log sequence         165
Current log sequence                    167

SQL> select name, log_mode from v$database;

NAME                   LOG_MODE
---------                                 ------------
TEST                       NOARCHIVELOG

SQL> show parameter spfile

NAME TYPE VALUE
------------------------------------ ----------- ------------------------------
spfile string /home/oracle/product/11g/db/db
s/spfiletest.ora

Setup Archive mode and archive destination:

ALTER SYSTEM SET log_archive_start=TRUE SCOPE=spfile;
ALTER SYSTEM SET log_archive_dest_1='location=/u01/oradata/orcl/archive/' SCOPE=spfile;
(Optionally we can also give other parameters in this statement like optional, reopen etc.)
ALTER SYSTEM SET log_archive_format='arch_%r_%t_%s.arc' SCOPE=spfile;

Now let shutdown the database and do a startup mount:

SHUTDOWN IMMEDIATE;
STARTUP MOUNT;
ARCHIVE LOG START;
ALTER DATABASE ARCHIVELOG;
ALTER DATABASE OPEN;

Test if archivelog is set properly:

SQL > archive log list

Database log mode Archive Mode
Automatic archival Enabled
Archive destination =/u01/oradata/orcl/archive/
Oldest online log sequence 165
Next log sequence to archive 167
Current log sequence 167

SQL>  select name, log_mode from v$database;

NAME    LOG_MODE
--------- ------------
TEST       ARCHIVELOG

SQL> alter system switch logfile;
System altered.

SQL> /
System altered.

Check physical archive log files:

SQL> !ls -lrt =/u01/oradata/orcl/archive/

-rw-r----- 1 oracle oracle 45509632 2009-03-20 14:43 1_167_677948664.dbf
-rw-r----- 1 oracle oracle 1024 2009-03-20 14:43 1_168_677948664.dbf

Archive log mode is now enabled and archive files started generating.



FOR RAC:

The ALTER DATABASE ARCHIVELOG command can only be performed if the database in mounted in exclusive mode. This means the whole clustered database must be stopped before the operation can be performed. First we set the relevant archive parameters:

ALTER SYSTEM SET log_archive_start=TRUE SCOPE=spfile;
ALTER SYSTEM SET log_archive_dest_1='location=/u01/oradata/MYSID/archive/' SCOPE=spfile;
ALTER SYSTEM SET log_archive_format='arch_%t_%s.arc' SCOPE=spfile;

Since we need to mount the database in exclusive mode we must also alter the following parameter:

ALTER SYSTEM SET cluster_database=FALSE SCOPE=spfile;


From the command line we can stop the entire cluster using:
srvctl stop database -d MYSID

With the cluster down we can connect to a single node and issue the following commands:

STARTUP MOUNT;
ARCHIVE LOG START;
ALTER DATABASE ARCHIVELOG;
ALTER SYSTEM SET cluster_database=TRUE SCOPE=spfile;
SHUTDOWN IMMEDIATE;

Notice that the CLUSTER_DATABASE parameter has been reset to it's original value. Since the datafiles and spfile are shared between all instances this operation only has to be done from a single node.

From the command line we can now start the cluster again using:
srvctl start database -d MYSID
The current settings place all archive logs in the same directory. This is acceptible since the thread (%t) is part of the archive format preventing any name conflicts between instances. If node-specific locations are required the LOG_ARCHIVE_DEST_1 parameter can be repeated for each instance with the relevant SID prefix.