Wednesday, October 28, 2009

Three new "hints"

In version 11.2.0.1, Oracle introduced three new hints: IGNORE_ROW_ON_DUPKEY_INDEX, CHANGE_DUPKEY_ERROR_INDEX and RETRY_ON_ROW_CHANGE. These are not normal hints like we know them. Oracle writes:

Note:
The CHANGE_DUPKEY_ERROR_INDEX, IGNORE_ROW_ON_DUPKEY_INDEX, and RETRY_ON_ROW_CHANGE hints are unlike other hints in that they have a semantic effect. The general philosophy explained in "Hints" does not apply for these three hints.

That's strange. Why does Oracle mix up the concept of a hint by introducing three new hints that have a semantic effect? Now you have two kind of hints. You have optimizer hints and you have semantic hints. Both are silently ignored when you make a syntax error. But there is no way to tell with which kind you are dealing, unless you know them by heart. And it becomes harder to explain. It means you cannot advice to just gather statistics and remove all hints anymore. Couldn't this be solved by adding an extra keyword? Or were they reluctant to introduce another deviation from the ANSI standard? On the other hand, we now have some extra options available, and that's good. Let's explore them.


IGNORE_ROW_ON_DUPKEY_INDEX

From the documentation: "When a unique key violation is encountered, a row-level rollback occurs and execution resumes with the next input row."

An example:

rwijk@ORA11GR2> create table mytable
2 ( id number(2) primary key
3 , name varchar2(3) not null
4 )
5 /

Table created.

rwijk@ORA11GR2> insert into mytable values (4,'old')
2 /

1 row created.

rwijk@ORA11GR2> insert into mytable values (9,'old')
2 /

1 row created.

rwijk@ORA11GR2> select * from mytable
2 /

ID NAM
---------- ---
4 old
9 old

2 rows selected.

rwijk@ORA11GR2> insert /*+ ignore_row_on_dupkey_index(mytable(id)) */ into mytable
2 ( id
3 , name
4 )
5 select level
6 , 'new'
7 from dual
8 connect by level <= 10
9 /

8 rows created.

rwijk@ORA11GR2> select *
2 from mytable
3 order by id
4 /

ID NAM
---------- ---
1 new
2 new
3 new
4 old
5 new
6 new
7 new
8 new
9 old
10 new

10 rows selected.

So the insert statement with the ignore_row_on_dupkey_index was specified to insert 10 rows, but due to the current contents of the table, only 8 rows were actually inserted, since 4 and 9 already exist. In 10g Release 2 you could achieve a similar effect by using the DBMS_ERRLOG package. Using the DBMS_ERRLOG package is a bit more cumbersome and it is different since it logs ALL errors, not just the unique key violations.


CHANGE_DUPKEY_ERROR_INDEX

From the documentation: "The CHANGE_DUPKEY_ERROR_INDEX hint provides a mechanism to unambiguously identify a unique key violation for a specified set of columns or for a specified index. When a unique key violation occurs for the specified index, an ORA-38911 error is reported instead of an ORA-001.". An example with a table containing a primary key constraint and a unique key constraint:

rwijk@ORA11GR2> create table t
2 ( id number(2)
3 , name varchar2(3) not null
4 , constraint t_pk primary key (id)
5 , constraint t_uk unique(name)
6 )
7 /

Table created.

rwijk@ORA11GR2> insert into t values (1,'bla')
2 /

1 row created.


Whether the primary key constraint or the unique key constraint is violated, you'll always get an ORA-00001 error message:

rwijk@ORA11GR2> insert into t values (1,'abc')
2 /
insert into t values (1,'abc')
*
ERROR at line 1:
ORA-00001: unique constraint (RWIJK.T_PK) violated


rwijk@ORA11GR2> insert into t values (2,'bla')
2 /
insert into t values (2,'bla')
*
ERROR at line 1:
ORA-00001: unique constraint (RWIJK.T_UK) violated



With the change_dupkey_error_index hint you can reroute one of the ORA-00001 error messages to an ORA-38911 error. Which unique key violation is rerouted, depends on the columns or unique index name specified after the table name in the hint.

rwijk@ORA11GR2> insert /*+ change_dupkey_error_index (t(name)) */ into t
2 values (2,'bla')
3 /
insert /*+ change_dupkey_error_index (t(name)) */ into t
*
ERROR at line 1:
ORA-38911: unique constraint (RWIJK.T_UK) violated


rwijk@ORA11GR2> insert /*+ change_dupkey_error_index (t(id)) */ into t
2 values (1, 'abc')
3 /
insert /*+ change_dupkey_error_index (t(id)) */ into t
*
ERROR at line 1:
ORA-38911: unique constraint (RWIJK.T_PK) violated


This can be handy in PL/SQL to distinguish one unique key violation from the other, without having to parse the error message for the constraint name:

rwijk@ORA11GR2> declare
2 e_name_uk_violated exception;
3 pragma exception_init(e_name_uk_violated,-38911);
4 begin
5 insert /*+ change_dupkey_error_index (t,t_uk) */ into t values (1,'abc');
6 exception
7 when e_name_uk_violated then
8 dbms_output.put_line('Name should be unique, and this name already exists.');
9 when dup_val_on_index then
10 dbms_output.put_line('Some other unique constraint was violated');
11 end;
12 /
Some other unique constraint was violated

PL/SQL procedure successfully completed.

rwijk@ORA11GR2> declare
2 e_id_pk_violated exception;
3 pragma exception_init(e_id_pk_violated,-38911);
4 begin
5 insert /*+ change_dupkey_error_index (t,t_pk) */ into t values (1,'abc');
6 exception
7 when e_id_pk_violated then
8 dbms_output.put_line('Id should be unique, and this Id already exists.');
9 when dup_val_on_index then
10 dbms_output.put_line('Some other unique constraint was violated');
11 end;
12 /
Id should be unique, and this Id already exists.

PL/SQL procedure successfully completed.



RETRY_ON_ROW_CHANGE

This one puzzles me. The documentation says: "When you specify this hint, the operation is retried when the ORA_ROWSCN for one or more rows in the set has changed from the time the set of rows to be modified is determined to the time the block is actually modified.".

Ok, I can reproduce that. First create a table t with 50,000 rows and add a row level trigger that logs autonomously into a table called logtable. This way we can monitor the progress of the update statement in a separate session. And it slows down the update considerably, which is a good thing for the test.

rwijk@ORA11GR2> create table t
2 ( id number(9) primary key
3 , col number(9) not null
4 )
5 /

Table created.

rwijk@ORA11GR2> insert /*+ append */ into t
2 select level
3 , level
4 from dual
5 connect by level <= 50000
6 /

50000 rows created.

rwijk@ORA11GR2> create table logtable
2 ( last_id number(9)
3 , number_of_processed_ids number(9)
4 )
5 /

Table created.

rwijk@ORA11GR2> insert into logtable values (null,0)
2 /

1 row created.

rwijk@ORA11GR2> create procedure p (p_id in t.id%type)
2 is
3 pragma autonomous_transaction;
4 begin
5 update logtable
6 set last_id = p_id
7 , number_of_processed_ids = number_of_processed_ids + 1
8 ;
9 commit;
10 end;
11 /

Procedure created.

rwijk@ORA11GR2> create trigger t_aru
2 after update on t
3 for each row
4 begin
5 p(:new.id);
6 end;
7 /

Trigger created.


And now we do an update with the retry_on_row_change hint:

rwijk@ORA11GR2> update /*+ retry_on_row_change */ t
2 set col = col * -1
3 /

And while this statement is running, open a second session and run this:

rwijk@ORA11GR2> select *
2 from logtable
3 /

LAST_ID NUMBER_OF_PROCESSED_IDS
---------- -----------------------
14754 14754

1 row selected.

rwijk@ORA11GR2> update t
2 set col = 888888
3 where id = 40000
4 /

1 row updated.

rwijk@ORA11GR2> commit
2 /

Commit complete.

rwijk@ORA11GR2> select *
2 from logtable
3 /

LAST_ID NUMBER_OF_PROCESSED_IDS
---------- -----------------------
17507 17508

1 row selected.

So the row with id 40000 is updated and committed, while our statement from session 1 has not reached this row yet.

After session 1 completed the update it says:

50000 rows updated.


But here is what happened (from session 1):

rwijk@ORA11GR2> select *
2 from t
3 where id between 39999 and 40001
4 /

ID COL
---------- ----------
39999 -39999
40000 -888888
40001 -40001

3 rows selected.

Session 2 has set COL to 888888, and session 1 has set this value to -888888.

rwijk@ORA11GR2> select *
2 from logtable
3 /

LAST_ID NUMBER_OF_PROCESSED_IDS
---------- -----------------------
50000 89922

1 row selected.

By the number_of_processed_ids column you can deduce that the statement was restarted once it has fetched the block with ID 40000 in it. I also checked it in session 2 by reexecuting the last statement several times. The last_id column dropped back to 1 and the entire statement was re-executed. This leads to a total of 89,922 (50,000 + almost 40,000) number of processed ID's. So the statement was retried on detecting a row change. And the table doesn't need the ROWDEPENDENCIES clause to make this happen.

But what's the point? Oracle has its write consistency mechanism that makes sure a statement gets restarted under the covers if during a DML a row has been changed that has not been locked yet. For example, if we add a where clause to the update statement saying "id = col", then the statement sets out to update 50,000 rows. Now, if another session update the col value of ID 40000 to 888888, the update statement of session 1 will detect that automatically and restarts the statement. Without the need for a RETRY_ON_ROW_CHANGE hint:

From session 1 with the same DDL as above:

rwijk@ORA11GR2> update t
2 set col = col * -1
3 where id = col
4 /

And while this statement runs, update the ID 40000 record in session 2:

rwijk@ORA11GR2> select *
2 from logtable
3 /

LAST_ID NUMBER_OF_PROCESSED_IDS
---------- -----------------------
10783 10783

1 row selected.

rwijk@ORA11GR2> update t
2 set col = 888888
3 where id = 40000
4 /

1 row updated.

rwijk@ORA11GR2> commit
2 /

Commit complete.

rwijk@ORA11GR2> select *
2 from logtable
3 /

LAST_ID NUMBER_OF_PROCESSED_IDS
---------- -----------------------
12822 12823

1 row selected.


And then from session 1:

49999 rows updated.

rwijk@ORA11GR2> commit
2 /

Commit complete.

rwijk@ORA11GR2> select *
2 from t
3 where id between 39999 and 40001
4 /

ID COL
---------- ----------
39999 -39999
40000 888888
40001 -40001

3 rows selected.

rwijk@ORA11GR2> select *
2 from logtable
3 /

LAST_ID NUMBER_OF_PROCESSED_IDS
---------- -----------------------
50000 89999

1 row selected.

So here Oracle's write consistency mechanism made the statement rollback and re-execute. Which, by the way, is a good reason to not use autonomous transactions in database triggers: they might execute twice for a single row.

If someone can enlighten me with a use case for the RETRY_ON_ROW_CHANGE hint, I would be happy.

Sunday, October 18, 2009

All About Grouping at Oracle OpenWorld

On Thursday 10AM I gave a presentation at the Unconference called "All About Grouping". I really need to think more about titles of presentations, as two people came in with the question "What kind of grouping will you be talking about?". Fortunately, when I mentioned it was about grouping in SQL and especially the extensions ROLLUP, CUBE and GROUPING SETS, they stayed.

I had given the presentation two times before, but this was the first time I did it in English. In fact, it was the first time I've presented anything in English. Things went well I guess: it looked like everybody understood what I was saying. And there was some nice interaction as well. So it was a good warm-up for the UKOUG conference.

Thanks to all who attended. You can download the presentation and the scripts below.

Powerpoint of All About Grouping

Demonstration scripts of All About Grouping

Friday, October 16, 2009

Oracle OpenWorld 2009, the last three days

Tuesday was a very rainy day. I decided to take the shuttle bus to the Moscone and was therefore a little late for the first session. The sessions I visited this day were:

Dan Morgan - What's New in Eleven ... Dot Two (that Oracle won't be talking about)
In every release there are like two types of features. On the one side there are features that are picked up by marketing and which are given catchy names. On the other side are the features that are not picked up by marketing. Dan talked about those. Some I recognized because they are mentioned in the New Features Guide, but he also mentioned a few others. Dan is involved in the beta program, so he has already worked with 11g Release 2 for a year now.

Richard Foote - Oracle Indexes: Q & A Discussion
Richard had only two slides which he showed in the first minutes, and then it was time to throw questions at him. About indexes of course, and David Bowie. Jonathan Lewis was also in the room, so they kind of answered the questions together. One of the things that was nice to hear, is that neither Richard nor Jonathan had seen a useful use for the skip scan. Which was something I already suspected.

Steven Feuerstein - Coding Therapy for Software Developers
Another great presentation by Steven. His presentation skills are simply awesome. The subject of the presentation helps of course. However, there were a few statements here that I strongly disagree with. Statements like "SQL statements with 3 table joins, or 7; that's crazy!" or "It doesn't make sense for developers to have to learn the ins and outs of tuning SQL!". Then who is going to write that transactional API or that PL/SQL interface or batch program? I suspect that Steven had some other context in mind when making those statements.

Raj Mattamal - Giving Your Oracle Application Express a Web 2.0 Face-Lift
Raj talked about how to make your APEX application cooler for the end user. With so much jquery code out there, it's easy to do more than just the standard APEX stuff. It makes a big difference in how you interact with the application. Raj did a live demonstration, which I always like. And, man, does Raj talk fast!

Andrew Holdsworth - Current Trends in Real-World Database Performance
This seems to be an annual session, but it was the first one I attended. It's nice to hear from the people who do performance stuff only, what they encounter most. It seems bad SQL is still number one.

Blogger Meetup
Alex Gorbachev organized the annual bloggers meeting this year. And he made it a special one with t-shirts saying "I met more bloggers than you", where everybody was writing their names on the t-shirts. The wearer of the shirt with the most names won a HP Netbook. This is a great event for meeting the people behind the blogs you read (or don't read yet).


Wednesday I really tried to get some more sleep. Again I was awake at 4 AM, about the same time as every morning here in San Francisco. But unfortunately just staying in bed until late in the morning did not help. The sessions I followed were:

Arup Nanda - Upgrade Case Study: Database Replay, Snapshot Standby, and Plan Baselines
This session mainly outlined the use of new technologies you can use for upgrading your database to 11g. Unfortunately for me, the technologies themselves were not explained here; it was about when to apply which technology.

Rory McLean - Using Row-Level Security with Oracle Application Express
Last year I swore not to enter sessions anymore when I don't know the speaker, but this session was only 30 minutes and there wasn't anything more interesting to choose from at that time, so I decided to give this one a shot. And I wasn't disappointed. Rory McLean gave a nice and well-prepared presentation, including a demonstration of VPD within APEX. I didn't learn much new though, but that's because I'm already familiar with the subject.

S. Gopalakrishnan - Keynote: Seven Game Changing Trends: How Prepared Are You?
I realized too late that, to have a good press/analysts/blogger seat for Larry's keynnote, you'd have to be in the room before this one started. When Andrew Clarke and I arrived at Moscone North's hall D, the room was already full, except for some seats at the side. So we ended up behind a giant pillar just watching the big screens. This first keynote was aimed at managers to give some high-level overview about the strategic challenges they are awaiting. Nothing much of interest to me.

Larry Ellison and others - Keynote
Larry's keynote did not have a big new introduction, as expected. He first mentioned the success of Oracle Enterprise Linux. Then he went on to talk about Exadata V2 for OLTP databases. Some impressive numbers were there. I think my current client should definitely check this one out. After they have migrated to 11g Release 2 of course ... And he repeated the $10M challenge and the attack on IBM, which he also did at the Sunday keynote. The third part was about introducing pro-active Oracle support. If you give Oracle all the data about your configuration, then Oracle will inform you about which patches you should run to avoid running into the same problems as other customers did. The fourth part was about the new Fusion Applications. This software is still not ready to ship, but it looked they are well on their way. Fusion Applications will be the successor of E-Business Suite, Siebel, Peoplesoft, JD Edwards and the like. Oracle will support those applications for a long long time, but they just offer an alternative based on modern technologies like SOA.

The highlight of the keynote was of course the appearance of the governor of California, Arnold Schwarzenegger. He did a very funny talk about the importance of technology. How it had helped him in his career and how important it is to reduce global warming.


The keynote was a long show that should have ended at 16:30, but ended at 17:20. I therefore missed "SQL gone wild: Taming Bad SQL the Easy Way (or the Hard Way)" by Benoit Dageville about tuning SQL with some automated tools.

Appreciation Event
Andrew invited me to join him and Marco Gralike, Jacco Landlust, Anjo Kolk, Tim Hall and Doug Burns to the Appreciation Event on Treasure Island. An evening with music performances by Aerosmith, Roger Daltrey and the Wailers and two other bands I did not see. Normally I don't fancy Aerosmith's music that much (with the exception of "Dream On"), but I thought they gave a great show. Roger Daltrey played a couple of songs I did not recognize, but when he played some of The Who classics, the enthusiasm of the crowd -and mine- was greatly increased. After losing everyone of the original group and having a peek at the Wailers, I called it a day.


I was back at the hotel at 1AM. And again awake at 4AM. My own Unconference session started at 10AM (more about that in a separate post). After my session I returned my laptop to my hotel and went back to see "Real-World Database Performance Roundtable" by five people, among whom Greg Rahn, Graham Wood and Andrew Holdsworth. It was a really nice Q&A session with some interesting questions from the audience. But I was so tired that when I closed my eyes for a moment, I actually started sleeping. Some internal mechanism woke me up instantly and I decided I had to leave before I started disrupting the session with some loud snoring ... I grabbed a lunch, had a quick talk with H.Tonguç Yılmaz and Husnu Sensoy (DBA of the year!), and took some photographs to include in a presentation about Oracle OpenWorld for my department. I went back to my hotel and slept from 2PM to 7PM. Finally ...

Tuesday, October 13, 2009

Oracle OpenWorld 2009, the first two days

This year I'm visiting Oracle OpenWorld for the second time. Everything is now familiar, except for two differences. First one is the location of Oracle Develop, which has moved back to the Hilton Hotel, instead of the nearby Marriot Hotel. It is a nice walk, but sometimes you cannot chat after a session if you have a next session at the Moscone Center. Second difference is that it seems to me less people are attending the conference than previous year. But it's still a lot of people. One thing hasn't changed: my excitement about the conference.

The first two days were very busy. I'll quickly describe the sessions I went to. Sunday started early at 8:30 AM.

Alex Gorbachev - Demystifying Oracle Real Application Clusters Workload
A topic that with which I am not too familiar: my shop doesn't use RAC and I'm not a DBA. So I thought to use this one to get to know a little bit more. I had some trouble getting the context right in the beginning of the session, but when Alex started demonstrating the stuff on his laptop, everything fell into place. The demos were simply great. The sometimes complex load balancing algorithms were made perfectly clear by showing exactly the right things. A good start of the conference.

Steven Feuerstein - Why you should care about PL/SQL in Oracle Database 11g Now!
This was the first time I attended a session by Steven Feuerstein. Although I heard nothing new about PL/SQL, it was great to see how he presented. Very interactive and with a lot of humour.

Dimitri Gielis - Mastering an Oracle Application Express Page
In this session we learnt about what happens when APEX shows or processes a page. The session really helped to get some of my thoughts straight. One of the new things I heard was a simple button on the page design screen called "View" where you can see the order of the events that will happen. I saw that page lots of times now, but never bothered to use that.

Andrew Clarke - Designing PL/SQL with Intent
This session was totally different than I expected. It was a rather philosophical one comparing design patterns in PL/SQL with design patterns in town planning. At the end a few PL/SQL examples were shown that I liked a lot, since I preach them myself often as well, although not as well as Andrew did.

Michael Hichwa & David Peake - Oracle Application Express 4.0
A lot of nice new features were presented from the APEX development team themselves. The ones I liked most were Dynamic Actions, with which you can create client side validations declaratively, and the APEX listener. I'm not entirely sure, but it looks like the APEX listener opens up easier and supported ways to have proxy database authentication. Other new features include integrated Team Development for feedback and bug reports and the like, improved charts, websheets applications and improved tabular form handling.

Bryn Llewellyn - Online Application Upgrade
This session was renamed to "Edition Based Redefinition". Bryn very concisely talked about the 11g Release 2's killer feature: editions. Without using a demo he managed to make it perfectly clear what the feature is about and what it is not about. A few random things I wrote down: from now on an object is not defined by owner and name but by owner, name and edition. Only synonyms, views and PL/SQL of all kinds are editionable. You need to rename the table, create an editioning view with the same name as the table's former name. Then move all triggers, all privileges and all VPD policies to the editioning view. And last but not least: in 11gR2 it's now possible to add an extra column while the table is in use, whereas previously you would get an ORA-04021 lock timeout message.

Scott McNealy - Extreme innovation
This keynote was mainly aimed at assuring all Sun customers that Oracle is not trying to kill Sun's products but instead Oracle is increasing the investments in Sun's products. Larry Ellison did a good job at depicting IBM's range of hardware as inferior, slow, and not very green. He even offers $10M if your Oracle application doesn't run at least twice as fast. Now that's confidence!

ACE dinner
An evening at the Sens Restaurant for all Oracle ACEs and ACE Directors, where I've met a lot of old and new friends. Thanks to Lillian and all others at OTN for the perfect organization and a nice evening!


On Monday I attended these sessions:

Jonathan Lewis - How to become an expert
A very entertaining story about what you should do to become an expert. My rather free interpretation of the answer to this question is, to learn and really understand the basics first. If you understand the architecture of version 6, when you could still carry the documentation with you, then everything afterwards is really just a variation on the same theme. Understand exactly what Oracle will have to do when you send it a SQL statement and you'll be in a good position to know where the root cause of your problem is.

Tom Kyte - All About Metadata: Why Telling the Database About Your Schema Matters
A good story about why should have constraints, even in data warehouses. They may slow down DML a little, but it's extra information for the cost based optimizer to make smarter decisions. Extra access paths will become available. A new thing for me was to see the effect of creating a dimension object in combination with materialized view query rewrites. I have to see that one for myself one time.

Greg Rahn - Chalk & Talk: The Core Performance Fundamentals Of Oracle Data Warehousing
A story about Exadata and data warehouses and benchmarks. A different world than my usual one, which was nice. For example, this exadata machine is so fast, Greg recommends to not even use indexes anymore in most data warehouses. Of course, it all depends. I must say I was impressed by all numbers he mentioned. Those are not numbers I usually see in my clients OLTP database.

Jonathan Lewis - How to Hint
Again a very good presentation in which I heard quite a number of new things. To name a few: use_hash is only half a hint; it's better to combine this hint with the swap_join_inputs hint to explicitly state which query block should act as the in-memory table and which one as the probe table. Jonathan also had a great example of the no_merge hint, which finally became clear to me. And the parallel hint was not what I thought it would be. It's not instructing to always go parallel, but rather: if you are full scanning the table AND it's cost is lower than the serial plan, only then go parallel. Which means there might be a day your query will run serial even if it ran in parallel before, or vice versa. In this session I also learnt how an alien looks like ...

Chen Shapira - Visualization Session - Make your charts more effective for troubleshooting and presenting
This one definitely categorizes as "something completely different". Although the running example was about duration of exports, the session was about how to use graphs effectively and how they can effectively support your message, or not. Luckily I knew just enough about statistics to understand all graphs. I especially liked the Q-Q one.

Lucas Jellema - Continuous Application Application Evolution with Oracle Database 11g Release 2
Exactly the same topic as Bryn Llewellyn's session yesterday. Lucas also had a clear story and even included a few live demos that really helped. A very well explained introduction. The only unfortunate thing about both sessions is that no downsides were mentioned. And there must be several. Like test efforts being multiplied or worse performance because crossedition triggers are row level triggers. And we know what that means ... But still I liked the session for being very clear and the nice metaphores.

OTN Night
Here I just grabbed some food, but I could not find any familiar faces so I decided to leave early and catch some sleep. Which doesn't work by the way: I'm still awake each morning well before 4AM. Which is not uncommon for Europeans, it seems ...

More to come later.

Monday, September 14, 2009

Check constraints and AND

Remember the opening statement of the MV error series? It was "Never put more than one error condition into one error message". Two months later I realized I didn't practice what I preach myself.

I'm currently busy developing a Tour de France application at home. The application has to store the standings. After each stage, a daily and overall standing becomes available for each of the following standing types: Individual, point, mountain and team. I'm storing standing lines and I chose a supertype implementation: all in one table. Which means the table looks something like this:

rwijk@ORA11GR1> create table standing_lines
2 ( stage_id number(6) not null
3 , standing_type varchar2(2) not null
4 , overall_or_daily varchar2(1) not null
5 , position number(3) not null
6 , rider_id number(6)
7 , team_id number(6)
8 , duration interval day(2) to second(0)
9 , number_of_points number(4)
10 , constraint pk primary key (stage_id,standing_type,overall_or_daily,position)
11 , constraint ck1 check (standing_type in ('IN','PO','MO','TE'))
12 , constraint ck2 check (overall_or_daily in ('D','O'))
13 )
14 /

Table created.

Note that I deliberately left out some unique, foreign key and other check constraints. For each standing type, different columns should be mandatory and other columns should stay null. For example: an individual standing line should contain a rider_id and a duration, and no team_id and no number_of_points. So it seems logical to code some additional check constraints like this:

rwijk@ORA11GR1> alter table standing_lines add constraint ck3 check
2 ( standing_type != 'IN'
3 or ( rider_id is not null
4 and team_id is null
5 and duration is not null
6 and number_of_points is null
7 )
8 )
9 /

Table altered.

rwijk@ORA11GR1> alter table standing_lines add constraint ck4 check
2 ( standing_type != 'PO'
3 or ( rider_id is not null
4 and team_id is null
5 and duration is null
6 and number_of_points is not null
7 )
8 )
9 /

Table altered.

rwijk@ORA11GR1> alter table standing_lines add constraint ck5 check
2 ( standing_type != 'MO'
3 or ( rider_id is not null
4 and team_id is null
5 and duration is null
6 and number_of_points is not null
7 )
8 )
9 /

Table altered.

rwijk@ORA11GR1> alter table standing_lines add constraint ck6 check
2 ( standing_type != 'TE'
3 or ( rider_id is null
4 and team_id is not null
5 and duration is not null
6 and number_of_points is null
7 )
8 )
9 /

Table altered.

But when I started coding routines to read some clob content pasted from the official Tour de France site, parsing all standing lines, and inserting the result into the standing_lines table, the inevitable occurred: it didn't work the first time. I received an error message like this:

ORA-02290: check constraint (RWIJK.CK6) violated

So I looked up the definition of this check constraint and all I could say was that I was trying to insert a team standing line and I gave rider_id a value, or I left team_id null, or I left duration null, or I gave number_of_points a value. That's four options. And it's certainly not one error condition per error message. To resolve this error, I first redefined my check constraints and split them into four each.

The first rewrite is by applying the distributivity rule: A or (B and C) == (A or B) and (A or C). In Oracle check constraint logic, it looks like this:

rwijk@ORA11GR1> alter table standing_lines drop constraint ck3
2 /

Table altered.

rwijk@ORA11GR1> alter table standing_lines drop constraint ck4
2 /

Table altered.

rwijk@ORA11GR1> alter table standing_lines drop constraint ck5
2 /

Table altered.

rwijk@ORA11GR1> alter table standing_lines drop constraint ck6
2 /

Table altered.

rwijk@ORA11GR1> alter table standing_lines add constraint ck3 check
2 ( ( standing_type != 'IN' or rider_id is not null )
3 and ( standing_type != 'IN' or team_id is null )
4 and ( standing_type != 'IN' or duration is not null )
5 and ( standing_type != 'IN' or number_of_points is null )
6 )
7 /

Table altered.

rwijk@ORA11GR1> alter table standing_lines add constraint ck4 check
2 ( ( standing_type != 'PO' or rider_id is not null )
3 and ( standing_type != 'PO' or team_id is null )
4 and ( standing_type != 'PO' or duration is null )
5 and ( standing_type != 'PO' or number_of_points is not null )
6 )
7 /

Table altered.

rwijk@ORA11GR1> alter table standing_lines add constraint ck5 check
2 ( ( standing_type != 'MO' or rider_id is not null )
3 and ( standing_type != 'MO' or team_id is null )
4 and ( standing_type != 'MO' or duration is null )
5 and ( standing_type != 'MO' or number_of_points is not null )
6 )
7 /

Table altered.

rwijk@ORA11GR1> alter table standing_lines add constraint ck6 check
2 ( ( standing_type != 'TE' or rider_id is null )
3 and ( standing_type != 'TE' or team_id is not null )
4 and ( standing_type != 'TE' or duration is not null )
5 and ( standing_type != 'TE' or number_of_points is null )
6 )
7 /

Table altered.

And now it becomes clear we can split them into four parts like this:

rwijk@ORA11GR1> alter table standing_lines drop constraint ck3
2 /

Table altered.

rwijk@ORA11GR1> alter table standing_lines drop constraint ck4
2 /

Table altered.

rwijk@ORA11GR1> alter table standing_lines drop constraint ck5
2 /

Table altered.

rwijk@ORA11GR1> alter table standing_lines drop constraint ck6
2 /

Table altered.

rwijk@ORA11GR1> alter table standing_lines add constraint ck3 check
2 ( standing_type != 'IN' or rider_id is not null )
3 /

Table altered.

rwijk@ORA11GR1> alter table standing_lines add constraint ck4 check
2 ( standing_type != 'IN' or team_id is null )
3 /

Table altered.

rwijk@ORA11GR1> alter table standing_lines add constraint ck5 check
2 ( standing_type != 'IN' or duration is not null )
3 /

Table altered.

rwijk@ORA11GR1> alter table standing_lines add constraint ck6 check
2 ( standing_type != 'IN' or number_of_points is null )
3 /

Table altered.

rwijk@ORA11GR1> alter table standing_lines add constraint ck7 check
2 ( standing_type != 'PO' or rider_id is not null )
3 /

Table altered.

rwijk@ORA11GR1> alter table standing_lines add constraint ck8 check
2 ( standing_type != 'PO' or team_id is null )
3 /

Table altered.

rwijk@ORA11GR1> alter table standing_lines add constraint ck9 check
2 ( standing_type != 'PO' or duration is null )
3 /

Table altered.

rwijk@ORA11GR1> alter table standing_lines add constraint ck10 check
2 ( standing_type != 'PO' or number_of_points is not null )
3 /

Table altered.

rwijk@ORA11GR1> alter table standing_lines add constraint ck11 check
2 ( standing_type != 'MO' or rider_id is not null )
3 /

Table altered.

rwijk@ORA11GR1> alter table standing_lines add constraint ck12 check
2 ( standing_type != 'MO' or team_id is null )
3 /

Table altered.

rwijk@ORA11GR1> alter table standing_lines add constraint ck13 check
2 ( standing_type != 'MO' or duration is null )
3 /

Table altered.

rwijk@ORA11GR1> alter table standing_lines add constraint ck14 check
2 ( standing_type != 'MO' or number_of_points is not null )
3 /

Table altered.

rwijk@ORA11GR1> alter table standing_lines add constraint ck15 check
2 ( standing_type != 'TE' or rider_id is null )
3 /

Table altered.

rwijk@ORA11GR1> alter table standing_lines add constraint ck16 check
2 ( standing_type != 'TE' or team_id is not null )
3 /

Table altered.

rwijk@ORA11GR1> alter table standing_lines add constraint ck17 check
2 ( standing_type != 'TE' or duration is not null )
3 /

Table altered.

rwijk@ORA11GR1> alter table standing_lines add constraint ck18 check
2 ( standing_type != 'TE' or number_of_points is null )
3 /

Table altered.

I reran my code and the error message became:

ORA-02290: check constraint (RWIJK.CK17) violated

Ah! I tried to insert a team standing line, but failed to give duration a value. Now that's more like it.

This is a very specific example, but you can say in general:
Whenever you have a check constraint and a top-level AND operator, you should split the constraint.

To know whether your expressions contains a top-level AND operator, it's essential you can calculate with boolean expressions. I mentioned the distributivity rule already above, and in that link you find some more useful rules. And I find I apply De Morgan's laws very often as well. These allow you to rewrite your expressions containing less mandatory brackets, so the AND and OR operators will go up in the hierarchy.

Whether you want to apply this rule for a BETWEEN operator (which also contains an AND :-) ) is subjective.

All About Grouping at OOW and UKOUG

My plan for 2009 was to try to do a presentation about grouping in SQL at both Oracle OpenWorld and at UKOUG. I submitted abstracts for both conferences. Unfortunately, the session was not selected for Oracle OpenWorld. Not very surprising really, since it is about functionality that was introduced in versions 8 and 9, so I can imagine other subjects are more important for Oracle. A bit later I read this post by Jonathan Lewis about the UKOUG conference, where Jonathan writes about more than 200 abstracts for 55 slots and a very high standard this year. So I started assuming that I would not be giving my presentation this year at all.

I did receive some good news though: CIBER management selected two people to visit Oracle OpenWorld, and I was one of them. Thanks Erik and Roel! A week later, I was totally surprised but delighted to receive an email telling that my session was selected for UKOUG. The agenda has been put online a couple of days ago, so you can see the proof here. And I was able to reserve a spot for Oracle OpenWorld's Unconference. So now I will be giving the presentation two times.

Details:
- Thursday 15th October 10 AM, at Oracle OpenWorld, Unconference, Moscone West Overlook I
- Wednesday 2nd December 13:50 - 14:50, at UKOUG Conference Series Technology & E-Business Suite, Track 3, Server Technology stream

You can read the UKOUG abstract here. If you are visiting one of those conferences, I hope to see you there.

Sunday, September 13, 2009

Fast refreshable materialized view errors, part seven: a summary

Previous posts:

Fast refreshable materialized view errors, part one (basis MV's)
Fast refreshable materialized view errors, part two: join MV's
Fast refreshable materialized view errors, part three: aggregate MV's
Fast refreshable materialized view errors, part four: union all MV's
Fast refreshable materialized view errors, part five: nested MV's
Fast refreshable materialized view errors, part six: MV_CAPABILITIES_TABLE


This post summarizes the previous posts by reversing the angle. You are developing fast refreshable materialized views and you experience a problem: either an error message appears when creating the MV or the MV gets created but just doesn't fast refresh. What should you do? Normally, the error message should tell you all about it, but as indicated in part one, this isn't always the case with materialized view errors. The list below will hopefully help in resolving your error.

This list may appear pretty complete after you've read all previous entries, but it is certainly not, due to several reasons:
- I could reproduce most error conditions, but not all.
- After having them reproduced, it's not certain the entire restriction was covered.
- There probably are a few undocumented restrictions. I've already encountered a few of them.
- The documentation has shown itself to be outdated and not entirely complete, so you cannot rely on that.
However, I hope to have covered at least the majority of error conditions, so there is a good chance your situation is covered.


ORA-12032: cannot use rowid column from materialized view log on "RWIJK"."MYEMP"

A rather cryptic way to say the rowid column is missing from the materialized view log. I only saw this restriction mentioned for aggregate MV's (restriction 5: All tables in the materialized view must have materialized view logs specified with ROWID), but it should be true as well for union all MV's and join MV's. The error message is quite clear, but if you'd do an explain_mview you'd see the even clear message "mv log must have ROWID" for the
REFRESH_FAST_AFTER_INSERT capability.


ORA-12033: cannot use filter columns from materialized view log on "RWIJK"."MYEMP"

You violated restriction 3 for aggregate MV fast refresh: the materialized view logs must contain all columns from the table referenced in the materialized view. Again I think the error message is worded a little poorly. The REFRESH_FAST_AFTER_INSERT capability says it better: mv log does not have all necessary columns.


ORA-12052: cannot fast refresh materialized view RWIJK.EMP_MV

You want to know why, so this error message is not helpful. You probably violated one of the three restrictions below:

1) Restriction 4 for basic MV fast refresh: It cannot contain an analytic function. The REFRESH_FAST_AFTER_INSERT capability says: window function in mv.

2) Restriction 5 for basic MV fast refresh: It cannot contain a MODEL clause. The REFRESH_FAST_AFTER_INSERT capability says: a SPREADSHEET clause is present.

3) Restriction 2 for join MV fast refresh: Rowids of all the tables in the FROM list must appear in the SELECT list of the query. The REFRESH_FAST_AFTER_INSERT capability says: the SELECT list does not have the rowids of all the detail tables.

So when you encounter an ORA-12052, do an explain_mview to really see why your MV cannot be fast refreshed.


ORA-12053: this is not a valid nested materialized view

And why is this not a valid nested materialized view? You probably violated restriction 1 for nested MV fast refresh:All parent and base materialized views must contain joins or aggregates. An explain_mview is not helpful here, as the REFRESH_FAST_AFTER_INSERT capability says: requirements not satisfied for fast refresh of nested mv. As could be seen in part five, this message means one of your parent MV's is a basic MV.


ORA-22818: subquery expressions not allowed here

You probably violated restriction 3 for basis MV fast refresh: It cannot contain a SELECT list subquery. A clear error message. An explain_mview cannot be done; it gives error message ORA-32342: The EXPLAIN_MVIEW facility failed to explain the materialized view statement.


ORA-23413: table "RWIJK"."MYDEPT" does not have a materialized view log

This error message cannot be more clear: you need materialized view logs on the base tables/MV's and they are missing. This error message corresponds with:
- Restriction 4 for join MV fast refresh: Materialized view logs must exist with rowids for all the base tables in the FROM list of the query.
- Restriction 2 for aggregate MV fast refresh: All tables in the materialized view must have materialized view logs
- Restriction 2 for nested MV fast refresh: All the underlying objects (materialized views or tables) on which the materialized view is defined must have a materialized view log.


ORA-32334: cannot create prebuilt materialized view on a table already referenced by a MV

You violated restriction 3 for nested MV fast refresh:You cannot create both a materialized view and a prebuilt materialized view on the same table. A clear error message, so it doesn't matter that this one cannot be further explained by explain_mview.


ORA-32401: materialized view log on "RWIJK"."MYEMP" does not have new values

You violated restriction 6 for aggregate MV fast refresh: All tables in the materialized view must have materialized view logs specified with INCLUDING NEW VALUES. If you do an explain_mview, the REFRESH_FAST_AFTER_INSERT capability says: mv log must have new values.


ORA-32412: encrypted column "DEPTNO" not allowed in the materialized view log

You violated restriction 4 for aggregate MV fast refresh: None of the columns in the base table, referred to in the materialized view log, can be encrypted.


The materialized view is created, but just doesn't fast refresh

You probably violated one of the four restrictions below:

1) Restriction 15 for aggregate MV fast refresh: If the materialized view has SUM(expr) but no COUNT(expr), then fast refresh is supported only on conventional DML inserts and direct loads. In this case the REFRESH_FAST_AFTER_ONETAB_DML capability says: SUM(expr) without COUNT(expr).

2) Restriction 16 for aggregate MV fast refresh: If the materialized view has no COUNT(*), then fast refresh is supported only on conventional DML inserts and direct loads. In this case, the REFRESH_FAST_AFTER_ONETAB_DML capability says: COUNT(*) is not present in the select list.

3) Restriction 17 for aggregate MV fast refresh: A materialized view with MAX or MIN is fast refreshable after delete or mixed DML statements if it does not have a WHERE clause. In this case the REFRESH_FAST_AFTER_ONETAB_DML capability says: mv uses the MIN or MAX aggregate functions.

4) Restriction 22 for aggregate MV fast refresh: Materialized aggregate views with outer joins are fast refreshable after conventional DML and direct loads, provided only the outer table has been modified. In this case the MV_CAPABILITIES_TABLE says everything is ok.

So by using the MV_CAPABILITIES_TABLE, there is a good chance you may find out more about this situation.


ORA-12054: cannot set the ON COMMIT refresh attribute for the materialized view

You probably violated one of the restrictions below:

  • Restriction 1 for basic MV fast refresh: The materialized view must not contain references to non-repeating expressions like SYSDATE and ROWNUM.
  • Restriction 7 for basic MV fast refresh: It cannot contain nested queries that have ANY, ALL, or NOT EXISTS.
  • Restriction 8 for basic MV fast refresh: It cannot contain a [START WITH ...] CONNECT BY clause.
  • Restriction 10 for basis MV fast refresh: ON COMMIT materialized views cannot have remote detail tables.
  • Restriction 5 for join MV fast refresh: You cannot create a fast refreshable materialized view from multiple tables with simple joins that include an object type column in the SELECT statement.
  • Restriction 6 for join MV fast refresh: ANSI joins are not possible
  • Restriction 8 for aggregate MV fast refresh: Only SUM, COUNT, AVG, STDDEV, VARIANCE, MIN and MAX are supported for fast refresh.
  • Restriction 10 for aggregate MV fast refresh: Aggregate functions must occur only as the outermost part of the expression. That is, aggregates such as AVG(AVG(x)) or AVG(x)+ AVG(x) are not allowed.
  • Restriction 12 for aggregate MV fast refresh: If VARIANCE(expr) or STDDEV(expr) is specified, COUNT(expr) and SUM(expr) must be specified. Oracle recommends that SUM(expr *expr) be specified.
  • Restriction 13 for aggregate MV fast refresh: The SELECT list must contain all GROUP BY columns.
  • Restriction 18 for aggregate MV fast refresh: For materialized views with CUBE, ROLLUP, grouping sets, or concatenation of them, the SELECT list should contain grouping distinguisher that can either be a GROUPING_ID function on all GROUP BY expressions or GROUPING functions one for each GROUP BY expression. For example, if the GROUP BY clause of the materialized view is "GROUP BY CUBE(a, b)", then the SELECT list should contain either "GROUPING_ID(a, b)" or "GROUPING(a) AND GROUPING(b)" for the materialized view to be fast refreshable.
  • Restriction 19 for aggregate MV fast refresh: For materialized views with CUBE, ROLLUP, grouping sets, or concatenation of them, GROUP BY should not result in any duplicate groupings. For example, "GROUP BY a, ROLLUP(a, b)" is not fast refreshable because it results in duplicate groupings "(a), (a, b), AND (a)".
  • Restriction 21 for aggregate MV fast refresh: The SELECT column in the defining query cannot be a complex expression with columns from multiple base tables. A possible workaround to this is to use a nested materialized view.
  • Restriction 23 for aggregate MV fast refresh: Materialized aggregate views with outer joins are fast refreshable after conventional DML and direct loads, provided unique constraints exist on the join columns of the inner join table.
  • Restriction 24 for aggregate MV fast refresh: If there are outer joins, all the joins must be connected by ANDs and must use the equality (=) operator.
  • Restriction 25 for aggregate MV fast refresh: Materialized views with named views or subqueries in the FROM clause can be fast refreshed provided the views can be completely merged.
  • Restriction 26 for aggregate MV fast refresh: It cannot contain a HAVING clause with a subquery.
  • Restriction 1 for union all MV fast refresh:The defining query must have the UNION ALL operator at the top level.
  • Restriction 3 for union all MV fast refresh:The SELECT list of each query must include a UNION ALL marker, and the UNION ALL column must have a distinct constant numeric or string value in each UNION ALL branch. Further, the marker column must appear in the same ordinal position in the SELECT list of each query block.
  • Restriction 5 for union all MV fast refresh:Insert-only aggregate materialized view queries are not supported for materialized views with UNION ALL.
  • Restriction 6 for union all MV fast refresh:Remote tables are not supported for materialized views with UNION ALL.
Good luck in determining which one applies to your situation :-)

This is horrible of course. The ORA-12054 basically means: sorry, you are out of luck. And don't bother trying to explain the materialized view, because that just leads to an "ORA-32342: The EXPLAIN_MVIEW facility failed to explain the materialized view statement" message. Of course it would be nice if Oracle provided one error message per violated restriction. We can but hope ...